Tomcat crashes with Out of Memory Error!

[RashMin Gadhavi]

Dear Experts,

I seek your advise/solution for one of my ongoing production problem.
My tomcat in production machine crashes abruptly with logs in "catalina.out" as "Out of Memory".
Here is some useful information for you.

OS:Red Hat Enterprise Linux Server release 5 (Tikanga)
Kernel \r on an \m

bash# uname -a
Linux xxxx 2.6.18-8.el5 #1 SMP Fri Jan 26 14:15:21 EST 2007 i686 i686 i386 GNU/Linux

Tomcat version:
Server version: Apache Tomcat/6.0.29
Server built: July 19 2010 1458
Server number: 6.0.0.29
OS Name: Linux
OS Version: 2.6.18-8.el5
Architecture: i386
JVM Version: 1.5.0_01-b08
JVM Vendor: Sun Microsystems Inc.

Tomcat CATALINA.SH settings:JAVA_OPTS="-Djava.awt.headless=true -Dfile.encoding=UTF8 -server -Xms2048m -Xmx2048m
-XX:NewSize=512m -XX:MaxNewSize=512m -XX:PermSize=512m
-XX:MaxPermSize=512m -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC"



Logs in "catalina.out"


Exception in thread "CompilerThread0" java.lang.OutOfMemoryError: requested 1356912 bytes for Chunk::new. Out of swap space?

1. Is there any thing to do with application?
2. Does it seems OS/Hardware problem? (By the way server has 3Gig RAM and its IBM Blade server)

Regards,
Rashmin Gadhavi

[William Brogden]

How long does it run before crashing? -or- How many requests processed before crashing?

If this was my problem I would be using the Tomcat Management App to frequently check on available memory, number of request threads running, number of sessions, etc.

Bill

[RashMin Gadhavi]

First of all, thanks a lot William for your response.

1. How long does it run before crashing? - Its random
2. How many requests processed before crashing - I have not measured yet (How to measure I do not know)

Also I logged in to Tomcat manager section and got following information as you suggested.

jk-8409
***************
Max threads: 200 Current thread count: 20 Current thread busy: 11
Max processing time: 4398274 ms Processing time: 5746.116 s Request count: 3223 Error count: 3 Bytes received: 2.24 MB Bytes sent: 58.79 MB

More detailed status of my application which is hosted in "webapps" directory of tomcat.
***************************************************************************
Start time: Thu Dec 15 06:17:30 EST 2011 Startup time: 43 ms TLD scan time: 19 ms
Active sessions: 44 Session count: 489 Max active sessions: 49 Rejected session creations: 0 Expired sessions: 445 Longest session alive time: 13358 s Average session alive time: 3212 s Processing time: -4397991 ms
JSPs loaded: 49 JSPs reloaded: 49


Do I have to run JProfiler? Since its a production machine..its difficult for me to run JProfiler though I can do that, it at all its required.


Regards,
Rashmin Gadhavi

[Rob Spoor]

There are two reasons you get out of memory errors:
1) You have a memory leak somewhere. You keep references longer than you should, possibly even indefinitely. JProfile could definitely help you find the causes.
2) Your server simply cannot handle the load.

I suspect the first, but you really would need to profile your application. Perhaps you can create a copy in a test environment. You probably won't see the errors (or at least not directly), but you should still be able to find memory leaks.

[William Brogden]

The Manager app should also show current memory use - for example, mine just reported:

Free memory: 56.88 MB Total memory: 123.75 MB Max memory: 123.75 MB

How much Free memory does your system show?

Bill

[RashMin Gadhavi]

William,

Here is the free memory information I got from my tomcat's manager application.

Free memory: 1701.19 MB Total memory: 2047.56 MB Max memory: 2047.56 MB

We have a very good IBM Blade server, but my application runs on a VM machine with given 3gig memory to VM software. I think it should be fine with tomcat.
Only 2 applications are loaded in tomcat's web apps.

Regards,
Rashmin Gadhavi

[William Brogden]

Good first step.

Now, keep monitoring - frequently check the Management app for number of requests, long running Threads and memory use.

The management app also reports the number of requests and number of HTTP errors - all important information for tracking down your problem.

If free memory gradually decreases as the number of requests goes up, that indicates a programmatic memory leak like Rob said.

Bill