• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Ron McLeod
  • Rob Spoor
  • Tim Cooke
  • Junilu Lacar
Sheriffs:
  • Henry Wong
  • Liutauras Vilda
  • Jeanne Boyarsky
Saloon Keepers:
  • Jesse Silverman
  • Tim Holloway
  • Stephan van Hulst
  • Tim Moores
  • Carey Brown
Bartenders:
  • Al Hobbs
  • Mikalai Zaikin
  • Piet Souris

memory trace of the server with 2 instances of tomcat 7 running

 
Ranch Hand
Posts: 52
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
hi all,

I have a linux server which is running 2 similar instances of Tomcats 7.

Each of the tomcat instances has the exact same web application (.war) deployed.

The application is not actively used by the users, but the behavior of the application on both the tomcat instances is looking very sluggish.

I checked the memory utilization of the server machine, which is as follows:

top - 12:47:20 up 182 days, 43 min, 1 user, load average: 0.09, 0.16, 0.15
Tasks: 226 total, 1 running, 225 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1%us, 0.2%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 15529M total, 11474M used, 4054M free, 485M buffers
Swap: 16383M total, 0M used, 16383M free, 9633M cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7866 root 0 -20 43036 12m 2640 S 12 0.1 2479:02 scopeux
1 root 20 0 10548 824 692 S 0 0.0 1:41.80 init
2 root 20 0 0 0 0 S 0 0.0 0:00.16 kthreadd
3 root 20 0 0 0 0 S 0 0.0 3:09.18 ksoftirqd/0
5 root 20 0 0 0 0 S 0 0.0 15:56.61 kworker/u:0

=========================

yyy@xxx:~> free -m
total used free shared buffers cached
Mem: 15529 11469 4059 0 485 9633
-/+ buffers/cache: 1350 14178
Swap: 16383 0 16383

======================

yyy@xxx:~> df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-rootvol 5.0G 1.4G 3.4G 29% /
udev 7.6G 220K 7.6G 1% /dev
tmpfs 24G 0 24G 0% /dev/shm
/dev/sda1 510M 48M 437M 10% /boot
/dev/mapper/vg00-homevol 992M 12M 929M 2% /home
/dev/mapper/vg00-tmpvol 2.0G 201M 1.7G 11% /tmp
/dev/mapper/vg00-varvol 7.9G 319M 7.2G 5% /var
/dev/mapper/vg00-crashvol 2.0G 68M 1.9G 4% /crashdump
/dev/mapper/vg00-auditvol 248M 11M 226M 5% /var/log/audit
/dev/mapper/vg00-optvol 12G 1.5G 9.8G 13% /opt
/dev/mapper/vg00-webdienste 99G 3.7G 95G 4% /webdienste

--

Does the information above provide any pointers that can be addressed further.

The strange thing is the used memory consumption of 11474M:

Mem: 15529M total, 11474M used, 4054M free, 485M buffers

I am also planning to have Lymbda probe installed on each of the tomcat instances to get more details.

Afterwords, I am also planning to restart the server machine.

The application works fine on the other earlier version (5) of Tomcat instance. It's just that we have now migrated the application to Tomcat 7 with Java 6 and experiencing this behavior.

The new server machine only running 2 tomcat 7 instances and nothing else.

Any help shall he appreciated.

thanks.


 
Saloon Keeper
Posts: 24310
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Well, you don't appear to be thrashing Virtual Memory and the system CPU load looks OK.

You might be having some sort of network problem such as perhaps a noisy node on the circuit that forces lots of re-transmissions.

Another possibility might be too-frequent garbage collection cycles - jprofile should be able to graph that.

Finally, I'd use a diagnosing web client such as FireBug to see what's happening when you make application requests and get responses. See what the timings are and if you are getting more data transmitted than can be easily sent in a short time.

Also, just to make sure it's Tomcat's problem, check to make sure that the delays aren't coming from issues with client-side JavaScript or page rendering speed.
 
Norman Meister
Ranch Hand
Posts: 52
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks for your answer, Tim.

What could be the reason for so much used memory here? It looks a bit odd when comparing used to free memory ration.

Mem: 15529M total, 11474M used, 4054M free, 485M buffers

Secondly, what could be an optimal JVM's memory allocation setting in this case as it is an essential part of getting the best performance out of Tomcat. For eliminating excessive use of garbage collection, i think starting the JVM with a higher maximum heap memory by using the -Xmx switch will decrease the frequency with which garbage collection occurs?

Thirdly, how about setting the initial heap memory equal to the maximum heap memory? Will that eliminate any need for the JVM to resize or reallocate the heap memory, leaving more to be used by other memory-intensive processes?

In short, what could be the idea values for JVM parameters in this case?

Thanks.


Tim Holloway wrote:Well, you don't appear to be thrashing Virtual Memory and the system CPU load looks OK.

You might be having some sort of network problem such as perhaps a noisy node on the circuit that forces lots of re-transmissions.

Another possibility might be too-frequent garbage collection cycles - jprofile should be able to graph that.

Finally, I'd use a diagnosing web client such as FireBug to see what's happening when you make application requests and get responses. See what the timings are and if you are getting more data transmitted than can be easily sent in a short time.

Also, just to make sure it's Tomcat's problem, check to make sure that the delays aren't coming from issues with client-side JavaScript or page rendering speed.

 
Tim Holloway
Saloon Keeper
Posts: 24310
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I don't have enough information to tell where all your RAM is going. About 1.4 the RAM in use is being used for I/O buffers, but without a "top" memory listing or something similar, I couldn't tell what apps are consuming the rest.

optimal JVM memory settings for Tomcat are entirely dependent on what the webapps need. I don't think you can get much to run in less than 256MB, but the upper limit is dictated by what the webapps themselves consume.

It's OK to set initial and max memory to the same value. Initial memory is what you "know" how much RAM the app needs, and max is when you're wrong for one reason or another - including bad initial estimate and unusually high system loads. As far as I know (too lazy to look it up), once the max memory is allocated, the JVM never lets go of it for the rest of its life. GC frequency shouldn't be affected, although if the initial estimate is too low, expect some minor overhead while it ratchets up to working levels. This is another thing that the jprofile monitor can graph for you.
 
Norman Meister
Ranch Hand
Posts: 52
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
HI Tim, thanks for answering.


Here is the “top” sorted by %Mem:
========================
Cpu(s): 0.1%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 15529M total, 11450M used, 4079M free, 491M buffers
Swap: 16383M total, 0M used, 16383M free, 9711M cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5268 root 20 0 1754m 387m 10m S 0 2.5 8:27.45 java
6545 root 20 0 1670m 387m 10m S 0 2.5 8:39.13 java
7317 root 20 0 90812 44m 6640 S 0 0.3 19:08.92 opcle
4130 root 20 0 306m 38m 3288 S 0 0.2 34:12.76 python
7363 root 20 0 153m 20m 9748 S 0 0.1 3:48.14 coda
7866 root 0 -20 43036 12m 2640 S 1 0.1 2559:57 scopeux
7256 root 20 0 151m 11m 7716 S 0 0.1 43:40.37 ovcd
7283 root 20 0 118m 10m 7508 S 0 0.1 0:07.10 ovconfd
25142 root 20 0 203m 10m 8448 S 0 0.1 2:48.53 opcmsga
7313 root 20 0 68144 9724 8056 S 0 0.1 14:43.00 opcmona
7264 root 20 0 166m 9384 5568 S 0 0.1 1:31.34 ovbbccb

...

Any pointers..?

Tim Holloway wrote:I don't have enough information to tell where all your RAM is going. About 1.4 the RAM in use is being used for I/O buffers, but without a "top" memory listing or something similar, I couldn't tell what apps are consuming the rest.

optimal JVM memory settings for Tomcat are entirely dependent on what the webapps need. I don't think you can get much to run in less than 256MB, but the upper limit is dictated by what the webapps themselves consume.

It's OK to set initial and max memory to the same value. Initial memory is what you "know" how much RAM the app needs, and max is when you're wrong for one reason or another - including bad initial estimate and unusually high system loads. As far as I know (too lazy to look it up), once the max memory is allocated, the JVM never lets go of it for the rest of its life. GC frequency shouldn't be affected, although if the initial estimate is too low, expect some minor overhead while it ratchets up to working levels. This is another thing that the jprofile monitor can graph for you.

 
Tim Holloway
Saloon Keeper
Posts: 24310
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hmmm. About 4GB of totally unused RAM, it appears.

At this point, I'd be inclined to suspect that either your webapps are running short but intense bouts of resource usage - either something really compute-heavy or relying on some external resource that has a long latency.

Probably the best approach would be to run a profiler on the webapp and see where most of its time is being spent.
 
Norman Meister
Ranch Hand
Posts: 52
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks, will install the JProfile on the server.

free -m
total used free shared buffers cached
Mem: 7986 6212 1774 0 767 2857


I am also planning on setting the setenv to:

export CATALINA_OPTS="$CATALINA_OPTS -Xms4096m"
export CATALINA_OPTS="$CATALINA_OPTS -Xmx4096m"

# Increase maximum perm size for web base applications to 4x the default amount
# http://wiki.apache.org/tomcat/FAQ/Memoryhttp://wiki.apache.org/tomcat/FAQ/Memory
export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxPermSize=512m"

export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParallelGC"
export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=1500"
export CATALINA_OPTS="$CATALINA_OPTS -XX:GCTimeRatio=9"
export CATALINA_OPTS="$CATALINA_OPTS -server"

# Disable remote (distributed) garbage collection by Java clients
# and remove ability for applications to call explicit GC collection
export CATALINA_OPTS="$CATALINA_OPTS -XX:+DisableExplicitGC"



Tim Holloway wrote:Hmmm. About 4GB of totally unused RAM, it appears.

At this point, I'd be inclined to suspect that either your webapps are running short but intense bouts of resource usage - either something really compute-heavy or relying on some external resource that has a long latency.

Probably the best approach would be to run a profiler on the webapp and see where most of its time is being spent.

 
Tim Holloway
Saloon Keeper
Posts: 24310
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The PermGen size isn't likely to affect performance. You generally know when it's too small when you get an out of PermGen Space exception, which is most likely to happen when you've got a complex application with stuff like ORM and lots of servlets. It also tends to show up when you repeatedly re-deploy a webapp without restarting Tomcat, since the old version's PermGen space isn't completely recovered.

Java 8 gets rid of the whole PermGen thing, incidentally.
 
Norman Meister
Ranch Hand
Posts: 52
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi All,

Since yesterday, the application performance on Tomcat AS 7 is again very sluggish:

ABCD@serverABCD:~> uname -a
Linux serverABCD 3.0.101-0.35-default #1 SMP Wed Jul 9 11:43:04 UTC 2014 (c36987d) x86_64 x86_64 x86_64 GNU/Linux
ABCD@serverABCD:~> date
Tue Sep 8 09:45:10 CEST 2015
ABCD@serverABCD:~> ps -ef |grep java |grep ABCD2
root 24359 1 0 Aug19 ? 01:01:04 /webdienste/java/jdk/ABCD2.intranet.ABCD.com/current/bin/java -Djava.util.logging.config.file=/webdienste/appserv-jsp/tomcat/ABCD2.intranet.ABCD.com/current/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Xms4096m -Xmx4096m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:MaxGCPauseMillis=1500 -XX:GCTimeRatio=9 -server -XX:+DisableExplicitGC -Djava.endorsed.dirs=/webdienste/appserv-jsp/tomcat/ABCD2.intranet.ABCD.com/current/endorsed -classpath /webdienste/appserv-jsp/tomcat/ABCD2.intranet.ABCD.com/current/bin/bootstrap.jar:/webdienste/appserv-jsp/tomcat/ABCD2.intranet.ABCD.com/current/bin/tomcat-juli.jar -Dcatalina.base=/webdienste/appserv-jsp/tomcat/ABCD2.intranet.ABCD.com/current -Dcatalina.home=/webdienste/appserv-jsp/tomcat/ABCD2.intranet.ABCD.com/current -Djava.io.tmpdir=/webdienste/appserv-jsp/tomcat/ABCD2.intranet.ABCD.com/current/temp org.apache.catalina.startup.Bootstrap start
ABCD@serverABCD:~>

ABCD2 instance runs since 19th August 2015 (and no reboot was performed recently).

Both the OS platform as well as the Web Components have been normal – yesterday and today – no suspicious load on server:

09:51:55 up 222 days, 21:47, 1 user, load average: 0.27, 0.21, 0.16
USER TTY LOGIN@ IDLE JCPU PCPU WHAT
ABCD pts/0 09:36 3.00s 0.01s 0.00s w

Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 15529M total, 15302M used, 226M free, 365M buffers
Swap: 16383M total, 86M used, 16297M free, 5868M cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24359 root 20 0 5399m 4.2g 10m S 1 27.8 61:05.45 java

---

Only point of concern is the used memory:

Mem: 15529M total, 15302M used, 226M free, 365M buffers

Any pointers?


Tim Holloway wrote:The PermGen size isn't likely to affect performance. You generally know when it's too small when you get an out of PermGen Space exception, which is most likely to happen when you've got a complex application with stuff like ORM and lots of servlets. It also tends to show up when you repeatedly re-deploy a webapp without restarting Tomcat, since the old version's PermGen space isn't completely recovered.

Java 8 gets rid of the whole PermGen thing, incidentally.

 
Tim Holloway
Saloon Keeper
Posts: 24310
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
At the OS level, you might want to set up an ongoing vmstat whose results are collected to a file and then plotted with gnuplot or something like that (assuming you'd like graphs instead of just raw numbers). VMStat can help detect "spikes" in resource demands over time, where "top" displays only the demand of the instant.

This looks like it might help you understand how vmstat is used: http://www.admin-magazine.com/HPC/Articles/Stat-like-Tools-for-Admins

At the JVM level, try running the jprofile tool to see how it's using memory internally. Considering all the grief, it might be worth spending some money on some commercial analysis tools.
 
Norman Meister
Ranch Hand
Posts: 52
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Attached are some stats. I will also try to get vmstat.


****

ABCD2:/webdienste/appserv-jsp/tomcat/ABCD2.com/current # ps aux | sort -k 5n
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 10 0.0 0.0 0 0 ? S Jan28 3:00 [ksoftirqd/1]
root 11 0.0 0.0 0 0 ? S Jan28 19:59 [kworker/0:1]
root 1154 0.0 0.0 0 0 ? S Jan28 0:00 [kauditd]
root 1177 0.0 0.0 0 0 ? SN Jan28 5:43 [kipmi0]
root 1193 0.0 0.0 0 0 ? S< Jan28 0:00 [kmpathd]
root 1194 0.0 0.0 0 0 ? S< Jan28 0:00 [kmpath_handlerd]
root 12 0.0 0.0 0 0 ? S Jan28 0:51 [watchdog/1]
root 12112 0.0 0.0 0 0 ? S Mar26 2:17 [kworker/7:2]
root 13 0.0 0.0 0 0 ? S Jan28 0:02 [migration/2]
root 1340 0.0 0.0 0 0 ? S Jan28 0:00 [kjournald]
root 1341 0.0 0.0 0 0 ? S Jan28 0:00 [kjournald]
root 1342 0.0 0.0 0 0 ? S Jan28 0:52 [kjournald]
root 1343 0.0 0.0 0 0 ? S Jan28 6:16 [kjournald]
root 1344 0.0 0.0 0 0 ? S Jan28 0:00 [kjournald]
root 1345 0.0 0.0 0 0 ? S Jan28 0:00 [kjournald]
root 1346 0.0 0.0 0 0 ? S Jan28 0:00 [kjournald]
root 1347 0.0 0.0 0 0 ? S Jan28 0:19 [kjournald]
root 139 0.0 0.0 0 0 ? S< Jan28 0:34 [lpfc_worker_0]
root 140 0.0 0.0 0 0 ? S Jan28 0:00 [scsi_eh_0]
root 141 0.0 0.0 0 0 ? S< Jan28 0:00 [scsi_wq_0]
root 142 0.0 0.0 0 0 ? S Jan28 17:08 [kworker/0:2]
root 143 0.0 0.0 0 0 ? S Jan28 1:09 [kworker/13:1]
root 144 0.0 0.0 0 0 ? S Jan28 1:15 [kworker/14:1]
root 145 0.0 0.0 0 0 ? S< Jan28 0:29 [lpfc_worker_1]
root 146 0.0 0.0 0 0 ? S Jan28 0:00 [scsi_eh_1]
root 147 0.0 0.0 0 0 ? S< Jan28 0:00 [scsi_wq_1]
root 148 0.0 0.0 0 0 ? S Jan28 2:03 [kworker/1:1]
root 149 0.0 0.0 0 0 ? S Jan28 1:56 [kworker/2:1]
root 15 0.0 0.0 0 0 ? S Jan28 1:56 [ksoftirqd/2]
root 155 0.0 0.0 0 0 ? S Jan28 2:02 [kworker/4:1]
root 156 0.0 0.0 0 0 ? S Jan28 2:22 [kworker/5:1]
root 157 0.0 0.0 0 0 ? S Jan28 1:21 [kworker/8:1]
root 158 0.0 0.0 0 0 ? S Jan28 1:27 [kworker/10:1]
root 160 0.0 0.0 0 0 ? S Jan28 1:19 [kworker/15:1]
root 16 0.0 0.0 0 0 ? S Jan28 0:46 [watchdog/2]
root 162 0.0 0.0 0 0 ? S Jan28 1:53 [kworker/3:1]
root 163 0.0 0.0 0 0 ? S Jan28 0:00 [scsi_eh_2]
root 164 0.0 0.0 0 0 ? S Jan28 0:02 [hpsa]
root 17 0.0 0.0 0 0 ? S Jan28 0:03 [migration/3]
root 18 0.0 0.0 0 0 ? S Jan28 2:25 [kworker/3:0]
root 189 0.0 0.0 0 0 ? S Jan28 1:16 [kworker/9:1]
root 190 0.0 0.0 0 0 ? S Jan28 1:25 [kworker/12:1]
root 19 0.0 0.0 0 0 ? S Jan28 2:53 [ksoftirqd/3]
root 1927 0.0 0.0 0 0 ? S Jan28 1:11 [flush-253:5]
root 1928 0.0 0.0 0 0 ? S Jan28 6:48 [flush-253:6]
root 199 0.0 0.0 0 0 ? S< Jan28 0:00 [kmpath_aluad]
root 20 0.0 0.0 0 0 ? S Jan28 0:47 [watchdog/3]
root 2 0.0 0.0 0 0 ? S Jan28 0:00 [kthreadd]
root 21 0.0 0.0 0 0 ? S Jan28 0:02 [migration/4]
root 214 0.0 0.0 0 0 ? S< Jan28 0:00 [kmpath_rdacd]
root 22 0.0 0.0 0 0 ? S Jan28 2:21 [kworker/4:0]
root 23 0.0 0.0 0 0 ? S Jan28 3:14 [ksoftirqd/4]
root 23243 0.0 0.0 0 0 ? S Aug03 0:26 [kworker/2:2]
root 234 0.0 0.0 0 0 ? S Jan28 1:12 [kworker/11:1]
root 24 0.0 0.0 0 0 ? S Jan28 0:48 [watchdog/4]
root 25 0.0 0.0 0 0 ? S Jan28 0:03 [migration/5]
root 26 0.0 0.0 0 0 ? S Jan28 2:21 [kworker/5:0]
root 27 0.0 0.0 0 0 ? S Jan28 3:30 [ksoftirqd/5]
root 28 0.0 0.0 0 0 ? S Jan28 0:48 [watchdog/5]
root 29 0.0 0.0 0 0 ? S Jan28 0:02 [migration/6]
root 3 0.0 0.0 0 0 ? S Jan28 4:11 [ksoftirqd/0]
root 31 0.0 0.0 0 0 ? S Jan28 5:47 [ksoftirqd/6]
root 32 0.0 0.0 0 0 ? S Jan28 0:47 [watchdog/6]
root 33 0.0 0.0 0 0 ? S Jan28 0:03 [migration/7]
root 34 0.0 0.0 0 0 ? S Jan28 3:04 [kworker/7:0]
root 34731 0.0 0.0 0 0 ? S 12:27 0:00 [flush-253:3]
root 35 0.0 0.0 0 0 ? S Jan28 6:29 [ksoftirqd/7]
root 3557 0.0 0.0 0 0 ? S< Jan28 0:00 [bond0]
root 36 0.0 0.0 0 0 ? S Jan28 0:47 [watchdog/7]
root 366 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/u:1]
root 37 0.0 0.0 0 0 ? S Jan28 0:58 [migration/8]
root 38 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/8:0]
root 39 0.0 0.0 0 0 ? S Jan28 0:41 [ksoftirqd/8]
root 40 0.0 0.0 0 0 ? S Jan28 0:45 [watchdog/8]
root 405 0.0 0.0 0 0 ? S Jan28 0:00 [khubd]
root 41 0.0 0.0 0 0 ? S Jan28 0:59 [migration/9]
root 42 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/9:0]
root 43 0.0 0.0 0 0 ? S Jan28 0:44 [ksoftirqd/9]
root 44 0.0 0.0 0 0 ? S Jan28 0:46 [watchdog/9]
root 45 0.0 0.0 0 0 ? S Jan28 0:58 [migration/10]
root 46 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/10:0]
root 47 0.0 0.0 0 0 ? S Jan28 0:41 [ksoftirqd/10]
root 48 0.0 0.0 0 0 ? S Jan28 0:46 [watchdog/10]
root 49 0.0 0.0 0 0 ? S Jan28 0:59 [migration/11]
root 50 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/11:0]
root 5 0.0 0.0 0 0 ? S Jan28 19:29 [kworker/u:0]
root 51 0.0 0.0 0 0 ? S Jan28 0:33 [ksoftirqd/11]
root 511 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 513 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 515 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 518 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 52 0.0 0.0 0 0 ? S Jan28 0:49 [watchdog/11]
root 521 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 525 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 529 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 53 0.0 0.0 0 0 ? S Jan28 0:59 [migration/12]
root 533 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 537 0.0 0.0 0 0 ? S< Jan28 0:00 [kdmflush]
root 54 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/12:0]
root 55 0.0 0.0 0 0 ? S Jan28 0:44 [ksoftirqd/12]
root 56 0.0 0.0 0 0 ? S Jan28 0:51 [watchdog/12]
root 57 0.0 0.0 0 0 ? S Jan28 1:00 [migration/13]
root 58 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/13:0]
root 59 0.0 0.0 0 0 ? S Jan28 0:44 [ksoftirqd/13]
root 59006 0.0 0.0 0 0 ? S Sep06 0:00 [kworker/6:2]
root 60 0.0 0.0 0 0 ? S Jan28 0:48 [watchdog/13]
root 6 0.0 0.0 0 0 ? S Jan28 0:02 [migration/0]
root 61 0.0 0.0 0 0 ? S Jan28 0:58 [migration/14]
root 62 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/14:0]
root 62990 0.0 0.0 0 0 ? S Jul30 0:11 [flush-253:8]
root 63 0.0 0.0 0 0 ? S Jan28 0:56 [ksoftirqd/14]
root 64 0.0 0.0 0 0 ? S Jan28 0:47 [watchdog/14]
root 65 0.0 0.0 0 0 ? S Jan28 1:00 [migration/15]
root 654 0.0 0.0 0 0 ? S Jan28 0:03 [kjournald]
root 66 0.0 0.0 0 0 ? S Jan28 0:00 [kworker/15:0]
root 67 0.0 0.0 0 0 ? S Jan28 0:56 [ksoftirqd/15]
root 68 0.0 0.0 0 0 ? S Jan28 0:46 [watchdog/15]
root 69 0.0 0.0 0 0 ? S< Jan28 0:00 [cpuset]
root 70 0.0 0.0 0 0 ? S< Jan28 0:00 [khelper]
root 7 0.0 0.0 0 0 ? S Jan28 0:46 [watchdog/0]
root 71 0.0 0.0 0 0 ? S< Jan28 0:00 [netns]
root 72 0.0 0.0 0 0 ? S Jan28 0:13 [sync_supers]
root 73 0.0 0.0 0 0 ? S Jan28 0:00 [bdi-default]
root 74 0.0 0.0 0 0 ? S< Jan28 0:00 [kintegrityd]
root 75 0.0 0.0 0 0 ? S< Jan28 0:00 [kblockd]
root 76 0.0 0.0 0 0 ? S< Jan28 0:00 [md]
root 77 0.0 0.0 0 0 ? S Jan28 0:00 [khungtaskd]
root 7723 0.0 0.0 0 0 ? S Sep04 0:04 [kworker/6:0]
root 79 0.0 0.0 0 0 ? S Jan28 0:00 [kswapd0]
root 80 0.0 0.0 0 0 ? SN Jan28 0:00 [ksmd]
root 8 0.0 0.0 0 0 ? S Jan28 0:03 [migration/1]
root 81 0.0 0.0 0 0 ? SN Jan28 0:28 [khugepaged]
root 82 0.0 0.0 0 0 ? S Jan28 0:00 [fsnotify_mark]
root 83 0.0 0.0 0 0 ? S< Jan28 0:00 [crypto]
root 87 0.0 0.0 0 0 ? S< Jan28 0:00 [kthrotld]
root 88 0.0 0.0 0 0 ? S< Jan28 0:00 [kpsmoused]
root 975 0.0 0.0 0 0 ? S Jan28 1:56 [kworker/1:2]
root 1858 0.0 0.0 4012 496 ? Ss Jan28 0:00 /sbin/acpid
root 8442 0.0 0.0 4504 676 tty1 Ss+ Jan28 0:00 /sbin/mingetty --noclear tty1
root 8443 0.0 0.0 4504 676 tty2 Ss+ Jan28 0:00 /sbin/mingetty tty2
root 8444 0.0 0.0 4504 676 tty3 Ss+ Jan28 0:00 /sbin/mingetty tty3
root 8445 0.0 0.0 4504 676 tty4 Ss+ Jan28 0:00 /sbin/mingetty tty4
root 8446 0.0 0.0 4504 676 tty5 Ss+ Jan28 0:00 /sbin/mingetty tty5
root 8447 0.0 0.0 4504 676 tty6 Ss+ Jan28 0:00 /sbin/mingetty tty6
XYZ 32034 0.0 0.0 4612 680 ? Ss 12:17 0:00 /usr/lib64/ssh/sftp-server
root 34828 0.0 0.0 4880 2468 ? S Apr29 0:33 /opt/uc4/uc4q/1/smgr/bin/../../agent/unix/bin/UCXJLI3
root 39577 0.0 0.0 4916 2376 ? S Jan28 3:50 /opt/uc4/uc4p/1/smgr/bin/../../agent/unix/bin/UCXJLI3
root 34826 0.0 0.0 4948 956 pts/0 R+ 12:27 0:00 ps aux
uuidd 7003 0.0 0.0 6072 0 ? Ss Jan28 0:00 /usr/sbin/uuidd -q -T 0
uc4sy 34825 0.0 0.0 6228 1248 ? S Apr29 0:40 /opt/uc4/uc4q/1/smgr/bin/ucybsmgr UC4Q1
uc4sy 7935 0.0 0.0 6228 816 ? S Jan28 1:07 /opt/uc4/uc4p/1/smgr/bin/ucybsmgr UC4P
root 4077 0.0 0.0 8372 4460 ? Ss Jan28 7:02 /sbin/haveged -w 1024 -v 1
root 7854 0.0 0.0 8560 644 ? Ss Jan28 0:00 /opt/perf/bin/ttd
root 4283 0.0 0.0 8772 536 ? Ss Jan28 32:58 /usr/sbin/irqbalance
root 2194 0.0 0.0 10160 4 ? Ss Jan28 0:00 /opt/hp/hp-health/bin/hp-asrd -p 1
root 2195 0.0 0.0 10292 468 ? S Jan28 3:01 /opt/hp/hp-health/bin/hp-asrd -p 1
root 1 0.0 0.0 10548 720 ? Ss Jan28 2:05 init [3]
XYZ 29516 0.0 0.0 13900 2664 pts/0 S 12:09 0:00 bash
root 29527 0.0 0.0 13908 2896 pts/0 S 12:09 0:00 bash
root 34827 0.0 0.0 13992 924 pts/0 S+ 12:27 0:00 sort -k 5n
root 8002 0.0 0.0 14100 3000 ? S Jan28 0:14 /opt/hps/inventory/python/bin/python2 /opt/hps/lw_agt/bin/lw_agt.py -f
root 4058 0.0 0.0 14436 592 ? S<sl Jan28 0:12 /sbin/audispd
root 7880 0.0 0.0 14620 2224 ? S<s Jan28 4:01 /opt/perf/bin/midaemon
root 7111 0.0 0.0 14800 556 ? Ss Jan28 0:20 /usr/sbin/cron
root 7124 0.0 0.0 16652 452 ? S Jan28 0:00 /usr/sbin/smartd
root 4173 0.0 0.0 16936 384 ? S< Jan28 0:00 /sbin/udevd --daemon
root 4174 0.0 0.0 16936 212 ? S< Jan28 0:00 /sbin/udevd --daemon
root 748 0.0 0.0 16940 460 ? S<s Jan28 0:00 /sbin/udevd --daemon
root 2061 0.0 0.0 18040 928 ? S Jan28 0:00 hald-runner
root 4133 0.0 0.0 18912 792 ? Ss Jan28 0:18 /sbin/rpcbind
root 5362 0.0 0.0 21292 640 ? S Jan28 1:00 cmaided -p 15 -s OK -l /var/log/hp-snmp-agents/cma.log
100 1871 0.0 0.0 21328 708 ? Ss Jan28 0:12 /bin/dbus-daemon --system
XYZ 29497 0.0 0.0 23160 2504 pts/0 Ss 12:09 0:00 -ksh
root 7205 0.0 0.0 23356 468 ? S Jan28 0:11 cmasm2d -p 30 -l /var/log/hp-snmp-agents/cma.log
root 5294 0.0 0.0 23360 508 ? S Jan28 0:35 cmasasd -p 15 -s OK -l /var/log/hp-snmp-agents/cma.log
root 5397 0.0 0.0 23364 584 ? S Jan28 0:35 cmascsid -p 15 -s OK -l /var/log/hp-snmp-agents/cma.log
root 5051 0.0 0.0 23624 964 ? S Jan28 19:04 cmahealthd -p 30 -s OK -t OK -i -l /var/log/hp-snmp-agents/cma.log
root 5052 0.0 0.0 23660 1080 ? S Jan28 11:38 cmastdeqd -p 30 -l /var/log/hp-snmp-agents/cma.log
root 6963 0.0 0.0 23804 356 ? S Jan28 0:00 /opt/hp/hpsmh/bin/rotatelogs /var/spool/opt/hp/hpsmh/logs/error_log 5M
root 6964 0.0 0.0 23804 356 ? S Jan28 0:00 /opt/hp/hpsmh/bin/rotatelogs /var/spool/opt/hp/hpsmh/logs/access_log 5M
root 5135 0.0 0.0 25588 1108 ? S Jan28 111:27 cmasm2d -p 30 -l /var/log/hp-snmp-agents/cma.log
root 5091 0.0 0.0 25980 776 ? S Jan28 4:45 cmathreshd -p 5 -s OK -l /var/log/hp-snmp-agents/cma.log
root 5164 0.0 0.0 26188 836 ? S Jan28 74:44 cmaperfd -p 30 -s OK -l /var/log/hp-snmp-agents/cma.log
root 4297 0.0 0.0 27112 372 ? Ss Jan28 0:00 /usr/sbin/mcelog --daemon --config-file /etc/mcelog/mcelog.conf
ntp 4980 0.0 0.0 28068 1276 ? Ss Jan28 9:42 /usr/sbin/ntpd -p /var/run/ntp/ntpd.pid -g -u ntp:ntp -i /var/lib/ntp -c /etc/ntp.conf
root 5067 0.0 0.0 28352 1344 ? S Jan28 33:52 cmahostd -p 15 -s OK -l /var/log/hp-snmp-agents/cma.log
root 4056 0.0 0.0 29648 340 ? S<sl Jan28 0:06 /sbin/auditd -s disable
101 2057 0.0 0.0 33920 2920 ? Ss Jan28 1:10 /usr/sbin/hald --daemon=yes
root 2156 0.0 0.0 34764 1648 ? S Jan28 0:00 hald-addon-input: Listening on /dev/input/event1
root 2175 0.0 0.0 34772 1868 ? S Jan28 0:00 /usr/lib64/hal/hald-addon-cpufreq
root 4119 0.0 0.0 35596 1656 ? Sl Jan28 0:00 /opt/opsware/agent/bin/python /opt/opsware/agent/pylibs/shadowbot/daemonbot.pyc --conf /etc/opt/opsware/agent/agent.args
101 2177 0.0 0.0 36520 1544 ? S Jan28 0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
root 5200 0.0 0.0 38332 972 ? Sl Jan28 0:54 cmapeerd -l /var/log/hp-snmp-agents/cma.log
root 7873 0.0 0.0 39296 5916 ? Ss Jan28 3:53 /opt/perf/bin/perfalarm
root 7084 0.0 0.0 40244 1996 ? Ss Jan28 0:31 /usr/lib/postfix/master
postfix 24260 0.0 0.0 42100 2288 ? S 11:50 0:00 pickup -l -t fifo -u
postfix 7110 0.0 0.0 42260 2104 ? S Jan28 0:05 qmgr -l -t fifo -u
root 7866 0.9 0.0 43036 11776 ? S<s Jan28 3043:56 /opt/perf/bin/scopeux
root 29525 0.0 0.0 43132 3312 pts/0 S 12:09 0:00 sudo /webdienste/../bin/su
root 5225 0.0 0.0 51104 2252 ? Sl Jan28 2:07 cmaeventd -p 15 -l /var/log/hp-snmp-agents/cma.log
root 4959 0.0 0.0 51420 928 ? Ss Jan28 0:00 /usr/sbin/sshd -o PidFile=/var/run/sshd.init.pid
root 5328 0.0 0.0 51524 2484 ? Sl Jan28 21:24 cmaidad -p 15 -s OK -l /var/log/hp-snmp-agents/cma.log
root 5259 0.0 0.0 53276 2376 ? Sl Jan28 98:42 cmafcad -p 15 -s OK -l /var/log/hp-snmp-agents/cma.log
root 7360 0.0 0.0 54532 7152 ? Sl Jan28 0:00 /opt/OV/lbin/eaagt/opcmsgi
root 7313 0.0 0.0 68144 7576 ? Sl Jan28 17:31 /opt/OV/lbin/eaagt/opcmona
root 41603 0.0 0.0 70572 6828 ? SNl Sep07 0:20 /usr/sbin/snmpd -r -A -LF i /var/log/net-snmpd.log -p /var/run/snmpd.pid
root 29526 0.0 0.0 75380 2580 pts/0 S 12:09 0:00 /webdienste/../bin/su
hpsmh 6968 0.0 0.0 85924 660 ? S Jan28 0:27 /opt/hp/hpsmh/sbin/hpsmhd -DSSL -f /opt/hp/hpsmh/conf/smhpd.conf
hpsmh 6969 0.0 0.0 85924 640 ? S Jan28 0:00 /opt/hp/hpsmh/sbin/hpsmhd -DSSL -f /opt/hp/hpsmh/conf/smhpd.conf
hpsmh 6970 0.0 0.0 85924 640 ? S Jan28 0:00 /opt/hp/hpsmh/sbin/hpsmhd -DSSL -f /opt/hp/hpsmh/conf/smhpd.conf
hpsmh 6971 0.0 0.0 85924 640 ? S Jan28 0:00 /opt/hp/hpsmh/sbin/hpsmhd -DSSL -f /opt/hp/hpsmh/conf/smhpd.conf
hpsmh 6972 0.0 0.0 85924 640 ? S Jan28 0:00 /opt/hp/hpsmh/sbin/hpsmhd -DSSL -f /opt/hp/hpsmh/conf/smhpd.conf
root 6962 0.0 0.0 85924 1220 ? Ss Jan28 4:41 /opt/hp/hpsmh/sbin/hpsmhd -DSSL -f /opt/hp/hpsmh/conf/smhpd.conf
root 5418 0.0 0.0 87688 2756 ? Sl Jan28 0:50 cmanicd
root 31939 0.0 0.0 92452 3708 ? Ss 12:17 0:00 sshd: XYZ [priv]
root 29403 0.0 0.0 92664 3736 ? Ss 12:09 0:00 sshd: XYZ [priv]
XYZ 29496 0.0 0.0 92664 2168 ? S 12:09 0:00 sshd: XYZ@pts/0
XYZ 32033 0.0 0.0 92808 2352 ? S 12:17 0:00 sshd: XYZ@notty
root 7317 0.0 0.2 96356 37492 ? Sl Jan28 23:34 /opt/OV/lbin/eaagt/opcle -std
root 2060 0.0 0.0 98600 1104 ? Ssl Jan28 0:03 /usr/sbin/console-kit-daemon
root 4091 0.0 0.0 100640 4300 ? SLl Jan28 10:01 /sbin/multipathd
root 7283 0.0 0.0 120892 10020 ? Sl Jan28 0:08 /opt/OV/lbin/conf/ovconfd
root 7342 0.0 0.0 121504 8060 ? Sl Jan28 19:13 /opt/OV/lbin/eaagt/opcacta
root 41261 0.0 0.0 122032 1280 ? Sl Jul15 0:05 /sbin/rsyslogd -c 5 -f /etc/rsyslog.conf
root 7256 0.0 0.0 154744 9240 ? Ssl Jan28 51:58 /opt/OV/bin/ovcd
root 1970 0.0 0.0 160808 864 ? Ssl Jan28 39:03 hpasmlited -f /dev/hpilo
root 7363 0.0 0.1 161408 20296 ? Sl Jan28 4:32 /opt/OV/lbin/perf/coda
root 7264 0.0 0.0 170332 8612 ? Sl Jan28 1:48 /opt/OV/bin/ovbbccb -nodaemon
root 6982 0.0 0.0 199220 1184 ? Ssl Jan28 5:29 /usr/sbin/nscd
root 25142 0.0 0.0 208656 10604 ? Sl Jun07 4:33 /opt/OV/lbin/eaagt/opcmsga
root 4130 0.0 0.2 313492 35972 ? Sl Jan28 40:40 /opt/opsware/agent/bin/python /opt/opsware/agent/pylibs/shadowbot/daemonbot.pyc --conf /etc/opt/opsware/agent/agent.args
root 24359 0.2 27.7 5528980 4413908 ? Sl Aug19 61:25 /webdienste/java/jdk/ABCD2.com/current/bin/java -Djava.util.logging.config.file=/webdienste/appserv-jsp/tomcat/ABCD2.com/current/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Xms4096m -Xmx4096m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:MaxGCPauseMillis=1500 -XX:GCTimeRatio=9 -server -XX:+DisableExplicitGC -Djava.endorsed.dirs=/webdienste/appserv-jsp/tomcat/ABCD2.com/current/endorsed -classpath /webdienste/appserv-jsp/tomcat/ABCD2.com/current/bin/bootstrap.jar:/webdienste/appserv-jsp/tomcat/ABCD2.com/current/bin/tomcat-juli.jar -Dcatalina.base=/webdienste/appserv-jsp/tomcat/ABCD2.com/current -Dcatalina.home=/webdienste/appserv-jsp/tomcat/ABCD2.com/current -Djava.io.tmpdir=/webdienste/appserv-jsp/tomcat/ABCD2.com/current/temp org.apache.catalina.startup.Bootstrap start
root 47962 0.2 27.7 5574472 4410676 ? Sl Aug12 77:49 /webdienste/java/jdk/ABCD2.com/current/bin/java -Djava.util.logging.config.file=/webdienste/appserv-jsp/tomcat/ABCD2.com/current/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Xms4096m -Xmx4096m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:MaxGCPauseMillis=1500 -XX:GCTimeRatio=9 -server -XX:+DisableExplicitGC -Djava.endorsed.dirs=/webdienste/appserv-jsp/tomcat/ABCD2.com/current/endorsed -classpath /webdienste/appserv-jsp/tomcat/ABCD2.com/current/bin/bootstrap.jar:/webdienste/appserv-jsp/tomcat/ABCD2.com/current/bin/tomcat-juli.jar -Dcatalina.base=/webdienste/appserv-jsp/tomcat/ABCD2.com/current -Dcatalina.home=/webdienste/appserv-jsp/tomcat/ABCD2.com/current -Djava.io.tmpdir=/webdienste/appserv-jsp/tomcat/ABCD2.com/current/temp org.apache.catalina.startup.Bootstrap start
ABCD2:/webdienste/appserv-jsp/tomcat/ABCD2.com/current #

****



Tim Holloway wrote:At the OS level, you might want to set up an ongoing vmstat whose results are collected to a file and then plotted with gnuplot or something like that (assuming you'd like graphs instead of just raw numbers). VMStat can help detect "spikes" in resource demands over time, where "top" displays only the demand of the instant.

This looks like it might help you understand how vmstat is used: http://www.admin-magazine.com/HPC/Articles/Stat-like-Tools-for-Admins

At the JVM level, try running the jprofile tool to see how it's using memory internally. Considering all the grief, it might be worth spending some money on some commercial analysis tools.

 
Norman Meister
Ranch Hand
Posts: 52
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Let's have a look at the JVM memory utilization stats with graph generated using PSI-Probe (attached below).

The strange thing is the PS Old gen memory utilization.

The JVM settings in Tomcat are as follows:

***

# discourage address map swapping by setting Xms and Xmx to the same value
export CATALINA_OPTS="$CATALINA_OPTS -Xms4096m"
export CATALINA_OPTS="$CATALINA_OPTS -Xmx4096m"

# Increase maximum perm size for web base applications to 4x the default amount
# http://wiki.apache.org/tomcat/FAQ/Memoryhttp://wiki.apache.org/tomcat/FAQ/Memory
export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxPermSize=512m"

# Reset the default stack size for threads to a lower value (by 1/10th original)
# By default this can be anywhere between 512k -> 1024k depending on x32 or x64
# bit Java version.
# http://www.springsource.com/files/uploads/tomcat/tomcatx-large-scale-deployments.pdf
# http://www.oracle.com/technetwork/java/hotspotfaq-138619.html
# Note - JDK 7 on RHEL needs minimum -Xss256k
#export CATALINA_OPTS="$CATALINA_OPTS -Xss192k"

# Oracle Java as default, uses the serial garbage collector on the
# Full Tenured heap. The Young space is collected in parallel, but the
# Tenured is not. This means that at a time of load if a full collection
# event occurs, since the event is a 'stop-the-world' serial event then
# all application threads other than the garbage collector thread are
# taken off the CPU. This can have severe consequences if requests continue
# to accrue during these 'outage' periods. (specifically webservices, webapps)
# [Also enables adaptive sizing automatically]
export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParallelGC"

# This is interpreted as a hint to the garbage collector that pause times
# of <nnn> milliseconds or less are desired. The garbage collector will
# adjust the Java heap size and other garbage collection related parameters
# in an attempt to keep garbage collection pauses shorter than <nnn> milliseconds.
# http://java.sun.com/docs/hotspot/gc5.0/ergo5.html
export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=1500"

# A hint to the virtual machine that it.s desirable that not more than:
# 1 / (1 + GCTimeRation) of the application execution time be spent in
# the garbage collector.
# http://themindstorms.wordpress.com/2009/01/21/advanced-jvm-tuning-for-low-pause/
export CATALINA_OPTS="$CATALINA_OPTS -XX:GCTimeRatio=9"

# The hotspot server JVM has specific code-path optimizations
# which yield an approximate 10% gain over the client version.
export CATALINA_OPTS="$CATALINA_OPTS -server"

# Disable remote (distributed) garbage collection by Java clients
# and remove ability for applications to call explicit GC collection
export CATALINA_OPTS="$CATALINA_OPTS -XX:+DisableExplicitGC"

# Uncomment to enable jconsole remote connection management.
# export CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote
# -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9011
# -Dcom.sun.management.jmxremote.local.only=false
# -Dcom.sun.management.jmxremote.authenticate=false"

# Check for application specific parameters at startup
if [ -r "$CATALINA_BASE/bin/appenv.sh" ]; then
. "$CATALINA_BASE/bin/appenv.sh"
fi

echo "Using CATALINA_OPTS:"
for arg in $CATALINA_OPTS
do
echo ">> " $arg
done
echo ""

echo "Using JAVA_OPTS:"
for arg in $JAVA_OPTS
do
echo ">> " $arg
done
echo "_______________________________________________"
echo ""

***

Tim Holloway wrote:At the OS level, you might want to set up an ongoing vmstat whose results are collected to a file and then plotted with gnuplot or something like that (assuming you'd like graphs instead of just raw numbers). VMStat can help detect "spikes" in resource demands over time, where "top" displays only the demand of the instant.

This looks like it might help you understand how vmstat is used: http://www.admin-magazine.com/HPC/Articles/Stat-like-Tools-for-Admins

At the JVM level, try running the jprofile tool to see how it's using memory internally. Considering all the grief, it might be worth spending some money on some commercial analysis tools.

JVM_memory_utilization.png
[Thumbnail for JVM_memory_utilization.png]
 
Tim Holloway
Saloon Keeper
Posts: 24310
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Unfortunately, the amount of work (and access) required for me to solve a problem like this is close to the limit I can do for free, so I'm mostly only able to recommend best practices.

On the profile, however, I'm not sure about the Eden space sawtooth pattern. Something like that is typical for a webapp, but I think that in this case it may be a little extreme.

More often than not, problems in the Tomcat JVM come from sloppy memory-management practices in webapps. However, you're not crashing due to Out-of-Memory (are you?) so the sawtooth extremes suggest that maybe something goes on that acquires a lot of memory and then lets go of it all at once. Hard to say for sure, since normal garbage collection works in much the same way, although modern-day JVMs tend to recover garbage incrementally so as not to drag down the apps while doing do. So from a memory standpoint it may be doing what it needs to do, but you'd have to consider the application architecture as to whether it needs to acquire (and release) such great amounts of memory.

Also, did you check the client side of the equation? I have had apps which had no server issues to speak of, but because of bad caching, they really dragged.
 
Consider Paul's rocket mass heater.
reply
    Bookmark Topic Watch Topic
  • New Topic