• Post Reply Bookmark Topic Watch Topic
  • New Topic

Application Unresponsive : Thread Dump Analysis  RSS feed

harshvardhan ojha
Ranch Hand
Posts: 157
Android Java MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have a java based webservice application written on ApacheCXF. We are running into issues where the application becomes completely unresponsive throwing 503 on every request . At this given point we have seen that Old Gen gets completely filled and all the application does is Full GC all the time from then. At this point of time I was trying to analyze both the thread and the heap dump . I have seen some AJP threads in waiting for the monitor as per the thread stat given below.

"ajp-8009-9--ADDADA121312" daemon prio=10 tid=0x00002aaae7ac4800 nid=0x2b84 in Object.wait() [0x00002aab16e72000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006c81449e8> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1621)
- locked <0x00000006c81449e8> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1646)
at java.lang.Thread.run(Thread.java:662)

I have lot of thread pools in the application so the below thread stat I feel is waiting for some tasks to start executing :-> this I guess is 97% of the total threads and it seem to me as normal.

"MyPool-pool-12-thread-10" prio=10 tid=0x000000005958c800 nid=0xaef waiting on condition [0x000000006f876000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c2c6a338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:315)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

The overall status of these threads is below

Overall Thread Count 2078
Overall Monitor Count 68
Number of threads waiting for a monitor 0
Number of threads locking a monitor 49

Number of threads sleeping on a monitor 2035
Number of deadlocks 0
Number of Monitors without locking threads 0

97% of all threads are sleeping on a monitor.
This might indicate they are waiting for some external resource (e.g. database) which is overloaded or not available or are just waiting to get to do something (idle threads). You should check the sleeping threads with a filter excluding all idle threads.

I am in the process of analyzing the thread dump but is there anything which can be sensed out of the above details.
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!