Help with thread dump

[Hemang Jani]

Hi, I'm new to java and need some help analyzing the thread dump.
Any help understanding is appreciated.

Most of them are 'waiting for lock' and couple 'runnable'. Thread dump taken during off hours, but jvm usage is very high. Wondering which thead is the culprit. This is an ongoing issue.
Thread Dump taken on Windows.

Thanks

            "[STANDBY] ExecuteThread: '39' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@1ae8926 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[STANDBY] ExecuteThread: '38' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@175826c WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[STANDBY] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@1ae428e WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[STANDBY] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@19382ba WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[STANDBY] ExecuteThread: '35' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@16b8c67 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@112a6ee WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@6c2cb WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '32' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@67120b WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '31' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@5c2253 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@13b7f64 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@bcaa5e WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@f9f7f WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[STANDBY] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@1a2e20 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '26' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@1f3f69 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[STANDBY] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@ff683a WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[STANDBY] ExecuteThread: '24' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@1cec1ff WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE
          
            	sun.management.ThreadImpl.getThreadInfo0(Native Method)
          
            	sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:142)
          
            	weblogic.platform.VM15.threadDumpAsString(VM15.java:22)
          
            	weblogic.platform.VM.threadDumpAsString(VM.java:124)
          
            	weblogic.t3.srvr.JVMRuntime.getThreadStackDump(JVMRuntime.java:113)
          
            	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          
            	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          
            	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          
            	java.lang.reflect.Method.invoke(Method.java:585)
          
            	weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:516)
          
            	com.sun.jmx.mbeanserver.DynamicMetaDataImpl.getAttribute(DynamicMetaDataImpl.java:96)
          
            	com.sun.jmx.mbeanserver.MetaDataImpl.getAttribute(MetaDataImpl.java:181)
          
            	com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:638)
          
            	com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:659)
          
            	weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$12.run(WLSMBeanServerInterceptorBase.java:320)
          
            	java.security.AccessController.doPrivileged(Native Method)
          
            	weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:318)
          
            	weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$12.run(WLSMBeanServerInterceptorBase.java:320)
          
            	java.security.AccessController.doPrivileged(Native Method)
          
            	weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:318)
          
            	weblogic.management.mbeanservers.internal.SecurityInterceptor.getAttribute(SecurityInterceptor.java:281)
          
            	weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor$5$1.run(AuthenticatedSubjectInterceptor.java:297)
          
            	java.security.AccessController.doPrivileged(Native Method)
          
            	weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor$5.run(AuthenticatedSubjectInterceptor.java:295)
          
            	weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
          
            	weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor.getAttribute(AuthenticatedSubjectInterceptor.java:290)
          
            	weblogic.management.jmx.mbeanserver.WLSMBeanServer.getAttribute(WLSMBeanServer.java:269)
          
            	javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1387)
          
            	javax.management.remote.rmi.RMIConnectionImpl.access$100(RMIConnectionImpl.java:81)
          
            	javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1247)
          
            	java.security.AccessController.doPrivileged(Native Method)
          
            	javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1350)
          
            	javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:599)
          
            	javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
          
            	weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:553)
          
            	weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:443)
          
            	weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
          
            	weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
          
            	weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:439)
          
            	weblogic.rmi.internal.BasicServerRef.access$300(BasicServerRef.java:61)
          
            	weblogic.rmi.internal.BasicServerRef$BasicExecuteRequest.run(BasicServerRef.java:983)
          
            	weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
          
            "[STANDBY] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@9f7c6f WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[STANDBY] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@1225fbd WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[STANDBY] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@f8bc17 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[STANDBY] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@1f7b0c7 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@43042d WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@807bbc WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@1d009d5 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@d75f7a WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@13dca05 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[STANDBY] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@63d61 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@2a4437 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[STANDBY] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@e8e3b0 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[STANDBY] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@7e24df WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[STANDBY] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@17d437c WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@1658dfe WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[STANDBY] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@59ed73 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@12c40f7 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@9dfefb WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "DynamicListenThread[Default]" RUNNABLE native
          
            	java.net.PlainSocketImpl.socketAccept(Native Method)
          
            	java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
          
            	java.net.ServerSocket.implAccept(ServerSocket.java:450)
          
            	java.net.ServerSocket.accept(ServerSocket.java:421)
          
            	weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:34)
          
            	weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:522)
          
            	weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:419)
          
            	weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:165)
          
            	java.lang.Thread.run(Thread.java:595)
          
            "weblogic.GCMonitor" waiting for lock java.lang.ref.ReferenceQueue$Lock@1bb8664 TIMED_WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
          
            	weblogic.platform.GCMonitorThread.waitForNotification(GCMonitorThread.java:88)
          
            	weblogic.platform.GCMonitorThread.run(GCMonitorThread.java:64)
          
            "FSCacheRefQueueThread" waiting for lock java.lang.ref.ReferenceQueue$Lock@74174d WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
          
            	java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
          
            	workshop.util.filesystem.FSCache$FSCacheRefQueueThread.run(FSCache.java:65)
          
            "[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.socket.utils.DynaQueue@1a5f1b3 TIMED_WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	weblogic.socket.utils.DynaQueue.getW(DynaQueue.java:237)
          
            	weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:278)
          
            	weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
          
            	weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
          
            "[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE native
          
            	java.net.SocketInputStream.socketRead0(Native Method)
          
            	java.net.SocketInputStream.read(SocketInputStream.java:129)
          
            	weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:826)
          
            	weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:792)
          
            	weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:283)
          
            	weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
          
            	weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
          
            "weblogic.store._WLS_ApolloManaged" WAITING
          
            	sun.misc.Unsafe.park(Native Method)
          
            	java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
          
            	java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
          
            	java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
          
            	weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
          
            	weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:573)
          
            	weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:621)
          
            	java.lang.Thread.run(Thread.java:595)
          
            "[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE native
          
            	java.net.SocketInputStream.socketRead0(Native Method)
          
            	java.net.SocketInputStream.read(SocketInputStream.java:129)
          
            	weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:826)
          
            	weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:792)
          
            	weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:283)
          
            	weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
          
            	weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
          
            "LDAPConnThread-1 ldap://0.0.0.0:8001" RUNNABLE native
          
            	java.net.SocketInputStream.socketRead0(Native Method)
          
            	java.net.SocketInputStream.read(SocketInputStream.java:129)
          
            	java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
          
            	java.io.BufferedInputStream.read(BufferedInputStream.java:235)
          
            	netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
          
            	netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:538)
          
            	java.lang.Thread.run(Thread.java:595)
          
            "LDAPConnThread-0 ldap://0.0.0.0:8001" RUNNABLE native
          
            	java.net.SocketInputStream.socketRead0(Native Method)
          
            	java.net.SocketInputStream.read(SocketInputStream.java:129)
          
            	java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
          
            	java.io.BufferedInputStream.read(BufferedInputStream.java:235)
          
            	netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
          
            	netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:538)
          
            	java.lang.Thread.run(Thread.java:595)
          
            "DoSManager" TIMED_WAITING
          
            	java.lang.Thread.sleep(Native Method)
          
            	com.octetstring.vde.DoSManager.run(DoSManager.java:433)
          
            "VDE Transaction Processor Thread" waiting for lock com.octetstring.vde.backend.standard.TransactionProcessor@19f52e7 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:367)
          
            	com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212)
          
            "[STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@19c0a9d WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
          
            "weblogic.store.WLS_DIAGNOSTICS" WAITING
          
            	sun.misc.Unsafe.park(Native Method)
          
            	java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
          
            	java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
          
            	java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
          
            	weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
          
            	weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:573)
          
            	weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:621)
          
            	java.lang.Thread.run(Thread.java:595)
          
            "weblogic.timers.TimerThread" waiting for lock weblogic.timers.internal.TimerThread@fbf5f8 TIMED_WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:275)
          
            "weblogic.time.TimeEventGenerator" waiting for lock weblogic.time.common.internal.TimeTable@1d17684 TIMED_WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:286)
          
            	weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:117)
          
            	java.lang.Thread.run(Thread.java:595)
          
            "[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE native
          
            	java.net.SocketInputStream.socketRead0(Native Method)
          
            	java.net.SocketInputStream.read(SocketInputStream.java:129)
          
            	weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:826)
          
            	weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:792)
          
            	weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:283)
          
            	weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
          
            	weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
          
            "Timer-1" waiting for lock java.util.TaskQueue@bc0334 TIMED_WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.util.TimerThread.mainLoop(Timer.java:509)
          
            	java.util.TimerThread.run(Timer.java:462)
          
            "Timer-0" waiting for lock java.util.TaskQueue@176558e WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	java.util.TimerThread.mainLoop(Timer.java:483)
          
            	java.util.TimerThread.run(Timer.java:462)
          
            "Signal Dispatcher" RUNNABLE
          
            null
          
            "Finalizer" waiting for lock java.lang.ref.ReferenceQueue$Lock@1eb003c WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
          
            	java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
          
            	java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
          
            "Reference Handler" waiting for lock java.lang.ref.Reference$Lock@15c6a47 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
          
            "main" waiting for lock weblogic.t3.srvr.T3Srvr@457bd0 WAITING
          
            	java.lang.Object.wait(Native Method)
          
            	java.lang.Object.wait(Object.java:474)
          
            	weblogic.t3.srvr.T3Srvr.waitForDeath(T3Srvr.java:730)
          
            	weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:380)
          
            	weblogic.Server.main(Server.java:67)
          

[Jeff Verdegan]

You can't really tell with any certainty from a thread dump which threads are eating up the most CPU. Statistically, if one or a couple threads are gobbling up the CPU, then it's more likely that those will be the ones showing as "Running" at the instant you do the thread dump, but there's no guarantee. The only way to know for sure is with a profiler. You can get a free trial or limited version of JProfiler, JProbe, OptimizeIt, or AppPerfect, or you can use jvisualvm, which comes with the JDK.

[Rohan Dhapodkar]

Hi Hemang,
Thread dumps obtain on windows/linux normally contain native thread id as nid= which is missing in your thread dump.
eg.

"pool-4-thread-3" prio=10 tid=0x09c41800 nid=0xd47 waiting on condition [0xa0bfe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0xabfac928> (a java.util.concurrent.SynchronousQueue$TransferStack)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:453)
	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:352)
	at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:636)

   Locked ownable synchronizers:
	- None

Once you have this native thread id in your java stack trace, you can use Process Explorer ( google it) tool, to identify which specific native thread of your java process is consuming the CPU. You can then correlate thread id from your native java process to your java stack trace thread (nid is hex value).

I learnt this in one of the training.

[Niranjan Bhat]

Also you can use one of the java profilers to profile your application.
In these profilers, you can checkout the thread which is running for a lot of time and taking the most of CPU or the thread which is blocking other threads.
I will recommend you to use yourkityourkit java profiler, it is very intuitive and easy to setup.

[Hemang Jani]

Thank you all for your attention.

Rohan - the nid and tid are available on Unix/Linux systems but not on thread dump taken in Windows environment


This issue is only in production environment and do not have previledge to install any profilers on the production servers.

Windows 32-bit Windows 2003
JDK 1.5

Thought to take heap dump with jmap, but jmap is not available for JDK 1.5 Windows.

Running out of ideas here.

[Jeff Verdegan]

Hemang Jani wrote:Thank you all for your attention.

This issue is only in production environment and do not have previledge to install any profilers on the production servers.

You can install a profiler in your dev or other environment and tell it to remotely connect to production.

You can use JMX to remotely query your production process for all kinds of stuff, including lots of thread and memory information, such as how much total CPU time a given thread has consumed (though I though a thread dump already showed that).

You can try to make Dev or QA more like production--more load or whatever.

Otherwise all I can think of is to take lots of heap dumps and see which threads are usually running.

And finally, failing all else, you can explain to your boss that no progress is being made this way, and the only way you have left to proceed toward a solution is to install a profiler in production.