• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Gradual increase in Resident Memory usage by Jboss(Java) process

 
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
We are facing an issue where the Resident memory of the Java process grows gradually. We have Xmx defined at 4096 MB and XX:MaxPermSize=1536m. The number of active threads ~1500 with an Xss of 256K defined.

When the application server(JBoss 6.1) starts the resident memory used is ~5.6GB(have been using top command to monitor it); it gradually grows(around 0.3 to 0.5 Gb per day) till it grows to ~7.4 Gb, when the kernel's OOM killer kills the process due to shortage of RAM space(The server has 9GB of RAM).

We have been regularly monitoring the thread dump- no suspect of a thread leak. We are still unable to figure out from where this extra memory is coming from.

Pmap output shows a number of Anon blocks(apart from the regular blocks for stack and heap), mostly in arenas of 64 Mb, which are un-accounted for in terms of memory usage by heap, perm gen & stacks.

In the heap dump we have also tried looking for DirectByteBuffers and sun.misc.Unsafe objects, which are generally used for non-heap memory allocation, but the number of objects as well as the memory capacity seem nominal. Is it possible that there can still be un- freed native memory even after these objects are GCed? Any other classes that may result in using up non-heap memory?

Our application does have native calls on its own, but it's possible that some third party libs have them.

Any ideas on what could be causing this? Anything other detail/tool that could further help debugging such an increase? Any known issue that we should look out for? Platform : Jboss 6.1 running on Centos 5.6.
 
Bartender
Posts: 10780
71
Hibernate Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Ashay Vedasen wrote:Any ideas on what could be causing this? Anything other detail/tool that could further help debugging such an increase? Any known issue that we should look out for? Platform : Jboss 6.1 running on Centos 5.6.


I'm not sure, but my understanding was that memory used by the JVM is never normally released back for general system use. And even if it was, it's likely to be in big (and very possibly unpredictable) "chunks" - But I have to admit I'm no expert in this; just an ex-jobbing sysadmin.

What you can do, however, is put a limit on the amount of memory that a JVM will use, so you always have room to run other non-Java processes.

Alternatively, if you're running one of the newer Linuxes (not sure about Centos), you could have the JVM run in its own VM, which you could then "throttle" with standard system commands.

Just some thoughts; not sure if it answers your question.

Winston
 
Bartender
Posts: 1166
17
Netbeans IDE Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You are more likely to get an authoritative analysis by going to the JBoss forums- http://www.jboss.org/forums/ .
 
Rancher
Posts: 1044
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Winston Gutkowski wrote:
I'm not sure, but my understanding was that memory used by the JVM is never normally released back for general system use.



Several years and versions ago I made an experiment to test this. Whether the JVM was willing to return not longer used memory to the operating system seemed to depend on the Java version and not on the operating system (Windows, Linux). Some version did, some did not.
 
Sunglasses. AKA Coolness prosthetic. This tiny ad doesn't need shades:
a bit of art, as a gift, the permaculture playing cards
https://gardener-gift.com
reply
    Bookmark Topic Watch Topic
  • New Topic