I'm in the process of resolving some memory allocation issues with a production server of mine and have come across some activity I can't account for. I'm running JBoss 7.1 under JVM v1.7.60.
Specifically, I've found that when the server's idled (no connections active, no scheduler running) I can watch the heap slowly grow from about 80 to 130MB, at which point GC runs and the allocation's reset back to ~80. Thankfully it seems like any used memory's successfully collected but it's really aggravating to see that there's this continuous allocation even when the server's sitting idle (not to mention that it adds 'noise' to the legitimate memory usage I'm trying to understand).
I'd wanted to see if I could get to the bottom of the matter, so I did a manual GC and took a heap dump, let usage go until ~120MB and then did another.
When doing so I found that the allocations referenced the Xerces XML parsing library.
VisualVM lets me look into the references for some of the Xerces objects, but I'm afraid I'm too much of a newbie to know what I actually do with this information. I guess I was hopeful that if I kept walking down the hierarchy of references that are shown I'd stumble upon the class that was actually kicking off the parsing operation.
The notion is that I can't know where to look next if I don't know what's actually causing the activity to take place, you know?
Also, one thought I'd had was that the parsing may be linked to deployments via annotations (which I don't use), so I tried switching that off but to no avail.
I guess my question to you fine folks are:
Has anyone dealt with this (or a similar) JBoss problem before and might have a tip about what's going on?
Failing that, is anyone familiar enough with VisualVM's Reference view or just JMX-inspection stuff in general that could help me understand what I should be doing next?
You'll have to take thread dumps when the memory usage starts increasing during the "idle" period. Take around 3-4 thread dumps with around 3-5 second interval between each of them and then analyze the thread dumps to understand what's going on during that period.
The details you see there are the result of doing dumps, my problem is that I'm unable to take the information those dumps reveal (Xerces is performing some sort of parsing operation) and use it to take action. Since it isn't clear to me what's causing the parsing (since it happens even when I don't deploy a .war I have to assume it's something JBoss does by itself), I don't know what to do to get it to stop.
The dumps you posted are of a different kind. They show you the memory snapshot but not what the threads are currently doing. Taking a few thread dumps with a gap of few seconds, when you start noticing this behaviour will show you exactly what code each of the thread is executing and what is triggering that code i.e. the complete stacktrace. The link I posted earlier will help you get the thread dumps.