• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Bear Bibeault
  • Ron McLeod
  • Jeanne Boyarsky
  • Paul Clapham
Sheriffs:
  • Tim Cooke
  • Liutauras Vilda
  • Junilu Lacar
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Tim Holloway
  • fred rosenberger
  • salvin francis
Bartenders:
  • Piet Souris
  • Frits Walraven
  • Carey Brown

Monitoring threads in WebSphere 3.5 ??

 
Greenhorn
Posts: 24
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi friends:
Actually, I have installed WebSphere 3.5 Standard in AIX 4.3.3. This perform well, but we need to know how many threads are used at the moment. To achieve it, we are using a program that perform a thread dump each two minutes and send us a message about the number of threads used.
But after certain time (i.e. two hours or two days) this process abort the WebSphere. In order to solve this problem we need to reboot the server.
I would like to know if I need to change some parameters in AIX or in WebSphere.
Thanks in advance
Jorge
 
Ranch Hand
Posts: 286
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Do a kill -3 on the PID of the appserver;
check for a javacoreXXXX.txt in the Websphere\Appserver\bin directory.
 
Jorge Chata
Greenhorn
Posts: 24
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Sorry about the hugh message, I am adding the Thread Dump message. Could somebody help me. If you can see something wrong, please let me know
Thanks
Jorge
----- Native Stack -----
at 0xd00059b4 in pthread_mutex_trylock
at 0xd00120fc in _pthread_event_wait
at 0xd000e0ec in _cond_wait_local
at 0xd000e568 in _cond_wait
at 0xd000f1bc in pthread_cond_wait
at 0xd35e1ab8 in condvarWait
at 0xd35e410c in sysMonitorWait
at 0xd3557580 in lkMonitorWait
at 0xd34e7208 in JVM_MonitorWait
at 0xd34fcc28 in sysInvokeNative
at 0xd34f7210 in mmisInvokeJniMethodHelper
at 0xd34edbdc in ThreadCPUTimeMillis
unable to backtrace through native code - iar 0x3001e360 not in text area (sp is 0x2ff20870)
----- Monitors Held -----
Monitor pool info:
Initial monitor count: 32
Minimum number of free monitors before expansion: 5
Pool will next be expanded by: 54
Current total number of monitors: 108
Current number of free monitors: 51
Monitor Pool Dump (inflated object-monitors):
sys_mon_t:0x3000d9a8 infl_mon_t: 0x3000d618:
java.lang.ref.Reference$Lock@8003B460/8003B468: <unowned>
Waiting to be notified:
"Reference Handler" (0x300daa48)
sys_mon_t:0x3000da28 infl_mon_t: 0x3000d634:
java.lang.ref.ReferenceQueue$Lock@8003B070/8003B078: <unowned>
Waiting to be notified:
"Finalizer" (0x300e46b8)
sys_mon_t:0x3000daa8 infl_mon_t: 0x3000d650:
com.ibm.ejs.oa.pool.ThreadPool$PooledThread@80033600/80033608: <unowned>
Waiting to be notified:
"Pooled ORB request dispatch WorkerThread" (0x308356d8)
sys_mon_t:0x3000db28 infl_mon_t: 0x3000d66c:
com.ibm.ejs.oa.pool.ThreadPool$PooledThread@80033598/800335A0: <unowned>
Waiting to be notified:
"Pooled ORB request dispatch WorkerThread" (0x308371d8)
sys_mon_t:0x3000dba8 infl_mon_t: 0x3000d688:
java.lang.Object@8025EC30/8025EC38: <unowned>
Waiting to be notified:
"Thread-0" (0x3096caf8)
sys_mon_t:0x3000df28 infl_mon_t: 0x3000d74c:
java.util.Vector@80244720/80244728: <unowned>
Waiting to be notified:
"Thread-1" (0x30c44a88)
sys_mon_t:0x3000dfa8 infl_mon_t: 0x3000d768:
javax.rmi.CORBA.KeepAlive@80031790/80031798: <unowned>
Waiting to be notified:
"Thread-2" (0x30c95ed8)
sys_mon_t:0x3000e228 infl_mon_t: 0x3000d7f4:
com.ibm.ejs.sm.server.ManagedServer@800F5160/800F5168: <unowned>
Waiting to be notified:
"P=914614 =0:CT" (0x3000f348)
sys_mon_t:0x3000e428 infl_mon_t: 0x3000d864:
com.ibm.servlet.util.AlarmManager@8078AAC0/8078AAC8: <unowned>
Waiting to be notified:
"Thread-4" (0x30b8f9e8)
sys_mon_t:0x3144aa68 infl_mon_t: 0x00000000:
java.lang.Class@800234C0/800234C8: <unowned>
Waiting to be notified:
"Worker#5" (0x3104dc88)
"Worker#4" (0x30fffac8)
"Worker#0" (0x30ffffd8)
"Worker#1" (0x31000238)
"Worker#8" (0x3104e3a8)
"Worker#7" (0x3104e148)
"Worker#6" (0x3104dee8)
"Worker#9" (0x3104e608)
"Worker#3" (0x3104da28)
to be continued ...
 
Jorge Chata
Greenhorn
Posts: 24
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Part 2 of 2
JVM System Monitor Dump (registered monitors):
Integer lock access-lock: <unowned>
Sleep lock: <unowned>
Waiting to be notified:
"SessionManagerServlet-InvalidationThread" (0x319ad3a8)
"Thread-5" (0x31892f18)
"Thread-6" (0x319bcd78)
Method trace lock: <unowned>
UTF8 Cache lock: <unowned>
Heap lock: <unowned>
Rewrite Code lock: <unowned>
Monitor Cache lock: owner "Worker#4" (0x30fffac8) 1 entry
JNI Pinning lock: <unowned>
JNI Global Reference lock: <unowned>
Classloader lock: <unowned>
Linking class lock: <unowned>
Binclass lock: <unowned>
Monitor Registry lock: owner "Worker#4" (0x30fffac8) 1 entry
Thread queue lock: owner "Worker#4" (0x30fffac8) 1 entry
Thread identifiers (as used in flat monitors):
ident 1f "Thread-6" (0x319bcd78) ee 0x319bccac
ident 1e "SessionManagerServlet-InvalidationThread" (0x319ad3a8) ee 0x319ad2dc
ident 1d "Thread-5" (0x31892f18) ee 0x31892e4c
ident 1c "Thread-4" (0x30b8f9e8) ee 0x30b8f91c
ident 1b "Worker#9" (0x3104e608) ee 0x3104e53c
...
ident 13 "Worker#0" (0x30ffffd8) ee 0x30ffff0c
ident 11 "Plugin Cfg Regen Scheduler" (0x306f6e38) ee 0x306f6d6c
ident 10 "P=914614 =0:StandardRT=2:LocalPort=34542:RemoteHost=adaix7:RemotePort=34549:" (0x30d089e8) ee 0x30d0891c
ident f "P=914614 =0:LT=1 ort=34547" (0x30cc01d8) ee 0x30cc010c
ident e "Thread-3" (0x30c96f38) ee 0x30c96e6c
ident d "Thread-2" (0x30c95ed8) ee 0x30c95e0c
ident c "Thread-1" (0x30c44a88) ee 0x30c449bc
ident b "P=914614 =0:StandardRT=1:LocalPort=34543:RemoteHost=128.1.6.36:RemotePort=900:" (0x30c09148) ee 0x30c0907c
ident a "P=914614 =0:LT=0 ort=34542" (0x30a08918) ee 0x30a0884c
ident 9 "P=914614 =0:StandardRT=0:LocalPort=34541:RemoteHost=128.1.6.36:RemotePort=34437:" (0x309fdaf8) ee 0x309fda2c
ident 8 "Thread-0" (0x3096caf8) ee 0x3096ca2c
ident 7 "Pooled ORB request dispatch WorkerThread" (0x308371d8) ee 0x3083710c
ident 6 "Pooled ORB request dispatch WorkerThread" (0x308356d8) ee 0x3083560c
ident 5 "Finalizer" (0x300e46b8) ee 0x300e45ec
ident 4 "Reference Handler" (0x300daa48) ee 0x300da97c
ident 3 "Signal dispatcher" (0x300da0c8) ee 0x300d9ffc
ident 2 "P=914614 =0:CT" (0x3000f348) ee 0x3000f27c
Java Object Monitor Dump (flat & inflated object-monitors):
java.lang.Class@800234C0/800234C8
locknflags 00150100 Flat locked by threadIdent 15. Entrycount 2
javax.rmi.CORBA.KeepAlive@80031790/80031798
locknflags 80000d00 Monitor inflated infl_mon 0x3000d768
com.ibm.ejs.sm.server.SeriousEventListener$DeliveryThread@800319A0/800319A8
locknflags 000c0000 Flat locked by threadIdent c. Entrycount 1
com.ibm.ejs.sm.server.ManagedServer@800F5160/800F5168
locknflags 80001200 Monitor inflated infl_mon 0x3000d7f4
java.net.PlainSocketImpl@8023AF58/8023AF60
locknflags 000e0000 Flat locked by threadIdent e. Entrycount 1
java.util.Vector@80244720/80244728
locknflags 80000c00 Monitor inflated infl_mon 0x3000d74c
java.net.PlainSocketImpl@80256278/80256280
locknflags 000f0000 Flat locked by threadIdent f. Entrycount 1
java.lang.Object@8025EC30/8025EC38
locknflags 80000500 Monitor inflated infl_mon 0x3000d688
java.net.PlainSocketImpl@80347400/80347408
locknflags 000a0000 Flat locked by threadIdent a. Entrycount 1
com.ibm.servlet.engine.webapp.WebAppRequestDispatcher@806B4798/806B47A0
locknflags 00140200 Flat locked by threadIdent 14. Entrycount 3
com.ibm.servlet.util.AlarmManager@8078AAC0/8078AAC8
locknflags 80001600 Monitor inflated infl_mon 0x3000d864
com.ibm.servlet.engine.webapp.WebAppRequestDispatcher@80AC7B20/80AC7B28
locknflags 001a0200 Flat locked by threadIdent 1a. Entrycount 3
...
com.ibm.servlet.engine.webapp.WebAppRequestDispatcher@80F0B600/80F0B608
locknflags 00130200 Flat locked by threadIdent 13. Entrycount 3
 
Jorge Chata
Greenhorn
Posts: 24
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Here I send more information.
Fri Mar 15 11:28:24 2002
SIGQUIT received at d0100e7c in /usr/lib/libpthreads.a.
java full version "JDK 1.1.8 IBM build a118-20001116 (JIT enabled: jitc V3.1-JDK1.1-20001116)"
args: /usr/jdk_base/bin/aix/native_threads/java Comunicador REMATES 1
Operating Environment
---------------------
Host: adaix7:128.1.6.36
OS Level: AIX 4.3.3.0
Processors -
Architecture: POWER_PC (impl: unknown, ver: unknown)
How Many: 4
Enabled: 4
User Limits (in bytes except for NOFILE and NPROC) -
RLIMIT_FSIZE: infinity
RLIMIT_DATA: 134217728
RLIMIT_STACK: 33554432
RLIMIT_CORE: 1073741312
RLIMIT_NOFILE: 2000
NPROC(max): 262144
Page Space (in blocks) -
/dev/hd6: size=131072, free=130266
Application Environment
-----------------------
Signal Handlers -
SIGQUIT: sysThreadIDump (libjava.a)
SIGILL: sysThreadIDump (libjava.a)
SIGTRAP: JITSigTrapHandler (libjitc.a)
SIGABRT: sysThreadIDump (libjava.a)
SIGFPE: sysThreadIDump (libjava.a)
SIGBUS: sysThreadIDump (libjava.a)
SIGSEGV: sysThreadIDump (libjava.a)
SIGSYS: sysThreadIDump (libjava.a)
SIGPIPE: ignored
SIGUSR2: sysThreadIDump (libjava.a)
SIGJVM1: sysThreadIHJVM1 (libjava.a)
Environment Variables -
_=/usr/jdk_base/bin/aix/native_threads/java
LANG=en_US
LOGIN=root
IMQCONFIGCL=/etc/IMNSearch/dbcshelp
PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin
LC__FASTMSG=true
IMQCONFIGSRV=/etc/IMNSearch
CGI_DIRECTORY=/var/docsearch/cgi-bin
CLASSPATH=.:/usr/jdk_base/classes:/usr/jdk_base/lib/classes.jar:/usr/jdk_base/lib/rt.jar:/usr/jdk_base/lib/i18n.jar:/usr/jdk_base/lib/classes.zip:/usr/jdk_base/lib/math.jar:/usr/jd k_base/lib/rmi-iiop.jar:/usr/jdk_base/lib/jndi.jar:/usr/jdk_base/lib/cosnaming.jar:/usr/jdk_base/lib/rmiregistry.jar:/usr/jdk_base/lib/providerutil.jar
LOGNAME=root
MAIL=/usr/spool/mail/root
LOCPATH=/usr/lib/nls/loc
USER=root
DOCUMENT_SERVER_MACHINE_NAME=localhost
AUTHSTATE=compat
SHELL=/bin/ksh
ODMDIR=/etc/objrepos
JAVA_HOME=/usr/jdk_base
DOCUMENT_SERVER_PORT=49213
XFILESEARCHPATH=/usr/jdk_base/lib/locale/%L/%T/%N%S:
HOME=/
TERM=ANSI
MAILMSG=[YOU HAVE NEW MAIL]
PWD=/usr/IBMWebAS/hosts/default_host/default_app/servlets
DOCUMENT_DIRECTORY=/usr/docsearch/html
TZ=EST5EDT
A__z=! LOGNAME
LIBPATH=/usr/jdk_base/lib/aix/native_threads
NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat
Loaded Libraries (sizes in bytes)
---------------------------------
/usr/jdk_base/lib/aix/native_threads/libnet.a
filesize: 32955
text start: d4216000
text size: 80bb
data start: 204bdb60
data size: 5b4
/usr/jdk_base/lib/aix/native_threads/libjitc.a
filesize: 1319144
text start: d40d3000
text size: 1420e8
data start: 2001fed0
data size: b458
/usr/lib/libiconv.a
filesize: 377832
text start: d0034100
text size: 13f3a
data start: 20011da0
data size: a574
/usr/lib/libi18n.a
filesize: 123742
text start: d002c100
text size: 7adc
data start: 2001d6d0
data size: 112c
/usr/lib/nls/loc/en_US
filesize: 14340
text start: d0026000
text size: 2d71
data start: d0029130
data size: 2324
/usr/lib/libcfg.a
filesize: 165290
text start: d005c100
text size: bfb1
data start: f00d0ec8
data size: 4178
/usr/lib/libodm.a
filesize: 269490
text start: d0049100
text size: 12969
data start: f00c9e70
data size: 5604
/usr/jdk_base/lib/aix/native_threads/libjava.a
filesize: 730472
text start: d4020000
text size: b2568
data start: f07a8138
data size: 52e28
/usr/lib/libpthreads.a
filesize: 713478
text start: d0001000
text size: 22b4
data start: f0082000
data size: 41004
/usr/lib/libpthreads.a
filesize: 713478
text start: d00f3000
text size: 1ed8a
data start: f0144000
data size: 4008
/usr/lib/libcrypt.a
filesize: 11167
text start: d00250f8
text size: 87a
data start: f0081528
data size: 13c
/usr/lib/libc.a
filesize: 6411931
text start: d0160720
text size: 1c2987
data start: f0000ca0
data size: 7f950

...
 
Jorge Chata
Greenhorn
Posts: 24
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
...
Current Thread Details
----------------------
"Finalizer thread" (TID:0x30030808, sys_thread_t:0x20070c38)
Native Thread State: ThreadID: 00000102 Reuse: 1 DAEMON RUNNING
Native Stack Data : base: 200dc050 pointer 200dae38 used(4632) free(253416)
----- Monitors held -----
----- Native stack -----
_pthread_ksleep
_pthread_event_wait
_cond_wait_local
_cond_wait
pthread_cond_wait
sysThreadBlockingSection
sysMonitorWait - waiting on Finalize me queue lock
finalizer_loop
sysThread_shell
_pthread_body
------ Java stack ------ () prio=1 *current thread*
----------------------------------------------------------------------

Total Thread Count:8
Active Thread Count:3
JNI Thread Count:0
Full thread dump:
"Async Garbage Collector" (TID:0x300307b8, sys_thread_t:0x200771e8)
Native Thread State: ThreadID: 00000203 Reuse: 1 DAEMON MONITOR WAIT
Native Stack Data : base: 201251c0 pointer 20124e88 used(824) free(257224)
----- Monitors held -----
----- Native stack -----
sysThreadBlockingSection
sysMonitorWait
sysThreadSleep
threadSleep
gc_loop
sysThread_shell
_pthread_body
------ Java stack ------ () prio=1
----------------------------------------------------------------------
"Finalizer thread" (TID:0x30030808, sys_thread_t:0x20070c38)
Native Thread State: ThreadID: 00000102 Reuse: 1 DAEMON RUNNING
Native Stack Data : base: 200dc050 pointer 200dade8 used(4712) free(253336)
----- Monitors held -----
----- Native stack -----
_pthread_ksleep
_pthread_event_wait
_cond_wait_local
_cond_wait
pthread_cond_wait
sysThreadBlockingSection
sysMonitorWait - waiting on Finalize me queue lock
finalizer_loop
sysThread_shell
_pthread_body
------ Java stack ------ () prio=1 *current thread*
----------------------------------------------------------------------
"main" (TID:0x30030858, sys_thread_t:0x20008a18)
Native Thread State: ThreadID: 00000001 Reuse: 1 USER PRIMORDIAL MONITOR WAIT
Native Stack Data : base: 2ff2277c pointer 2ff21bf4 used(2952) free(255096)
----- Monitors held -----
----- Native stack -----
sysThreadBlockingSection
sysMonitorWait
sysThreadSleep
threadSleep
java_lang_Thread_sleep
------ Java stack ------ () prio=5
Comunicador.main(Compiled Code)
----------------------------------------------------------------------

System Monitor Status
---------------------
Thread queue lock: unowned.
Name and type hash table lock: unowned.
String intern lock: unowned.
JNI pinning lock: unowned.
JNI global reference lock: unowned.
Zip lock: unowned.
BinClass lock: unowned.
Class loading lock: unowned.
Java stack lock: unowned.
Code rewrite lock: unowned.
Heap Lock: unowned.
Has finalization queue lock: unowned.
Finalize me queue lock: unowned.
"Finalizer thread" (0x30030808) waiting
Integer lock access-lock: unowned.
Monitor cache lock: unowned.
Monitor registry: unowned.
Object Monitor Status
---------------------
GC Suspension Points
--------------------
GC cycle -0 suspension points:
tid 0x200771e8: doing GC
tid 0x20070c38: in blocking section called from 0xd4020b4c (sysMonitorWait)
tid 0x20008a18: in blocking section called from 0xd4020c2c (sysMonitorWait)
GC cycle -1 suspension points:
tid 0x200771e8: doing GC
tid 0x20070c38: in blocking section called from 0xd4020b4c (sysMonitorWait)
tid 0x20008a18: in blocking section called from 0xd4020c2c (sysMonitorWait)
GC cycle -2 suspension points:
tid 0x2028b028: in blocking section called from 0xd4043750 (sysConnectFD)
tid 0x200771e8: doing GC
tid 0x20070c38: in blocking section called from 0xd4020b4c (sysMonitorWait)
tid 0x20008a18: in blocking section called from 0xd4020c2c (sysMonitorWait)
GC cycle -3 suspension points:
tid 0x2028b028: in blocking section called from 0xd4043750 (sysConnectFD)
tid 0x200771e8: doing GC
tid 0x20070c38: in blocking section called from 0xd4020b4c (sysMonitorWait)
tid 0x20008a18: in blocking section called from 0xd4020c2c (sysMonitorWait)
 
No matter. Try again. Fail again. Fail better. This time, do it with this tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
    Bookmark Topic Watch Topic
  • New Topic