JBoss 4.0.4 is slow down in every 20 days

[Sky Loi]

Hi, friends,

I have production env with JBoss 4.0.4 and was add ehcache in last release. BUt it was slow down and then shutdown due to the java heap out of memory in every 20 days after that and need reboot the jboss. So far, I have not found the root cause, just got some symptons: java heap usage is overflow, e.g. over 512 max to 6xx sometime, seems run out memory. But it is ok in the 19 days when I monitor the memory usage, usually have about 120 - 150M free memory left. Do you have any idea or experience on my case?

The specs:
JBoss 4.04
Java 1.5.0.07
server physical mem: 1G


Thanks

[Peter Johnson]

The best thing to do is add -XX:+HeapDumpOnOutOfMemoryError to the JAVA_OPTS in the run script. Then when you get the OOME, the JVM will dump the heap contents to a file. You can open the file using a tool such as VisualVM. That might tell you what is using up the heap.

You should also monitor GC statistics. You can usually tell, by monitoring the heap usage after a major collection, whether the app has a leak - the tend line will have a positive slope.
For suggestions on how to gather GC data and analyze it, see these white papers:
Java Garbage Collection Statistical Analysis 101
Java Garbage Collection Performance Analysis 201
at http://www.cmg.org/cgi-bin/search.cgi?q=java+peter+johnson&x=30&y=10

[Sky Loi]

Hi, Peter,

Thanks for your great advice and direction. I will try this way to find the root cause.

[Sky Loi]

I just got the heap memory issue. The rest heap free size was just less than 1M (pre-allocated 512M). I got the thread dump at that time. The attachment is the thread dump logs. I did not see much exception. One thing I remembered I add new ehcache/jgroup components in the JBoss which seems has some conflict with the built-in jboss clustering components (the old jgroup libs). I am not sure it is the clue I could start with.

[Sky Loi]

sorry, seems I get the thread dump instead of heap dump. I will check the heap dump way first.

[Sky Loi]

I used JProfiler to monitor the heap in JBoss server and found some suspicious java.lang.Object instances are increased gradually but cannot see what type exactly they are. I really assume it is the cause leading to 20-day heap memory leaking. Do we have any better idea how to get more into it? or any other way or tool?

Thanks