• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Heap Usage Drops Drastically After Heap Dump

 
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I had a process that I thought had a memory leak.  So using VisualVM I created a heap dump.  Surprisingly, that causes the heap usage to drop from about 300MB back down to it's expected level of 30MB.

Did the heap dump trigger some kind of GC?  And why didn't that GC occur automatically?  I attached the screenshot of what happened.
heap.png
[Thumbnail for heap.png]
 
Johnny Phan
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
This only occurred recently.  Before, I've had this process running for months on end without problem.  Recently the process started having OutOfMemory exceptions.

And just now I realized the only difference is I've recently switched to using Java 10 runtime.  I'm going to switch back to JDK8 soon see if the problem goes away.  Possibly that has something to do with it.
 
Johnny Phan
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Actually, I haven't recompiled the code using JDK10.  So maybe there's a problem with JDK10 running Java8-compiled byte code.

I will first recompile the code using JDK10 see if the problem will go away.
 
author
Posts: 67
10
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Just to reply to one of the points here - yes, using VisualVM to perform a Heap Dump will always cause a Full GC to be performed before the dump.

This is because a heap dump is an expensive operation (usually considered a last resort for production systems), so the reasoning is that it's acceptable to Full GC first in order to minimize the size of the heap dump produced, by removing dead objects.

So, by triggering a heap dump, you are forcing a Full GC - regardless of whether the heap needs one or not. That is why the GC didn't occur automatically - it did not need to run, but you forced it to.

It is very unlikely that there is a regression or other problem with JDK 10 running JDK 8 bytecode. Far more likely is that a small a change in your code has caused more objects to survive into old gen (increasing pressure on the GC) or that you have accidentally introduced a memory leak.

There is an old saying: "Suspecting a platform (or compiler) bug in a mature system is the last resort, not the first"
 
Saloon Keeper
Posts: 27762
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I'm pretty sure that all the way back in Java 1.4, doing a heap dump would initiate garbage collection as a property of the JVM, and not exclusive to any particular external diagnostic tool (that is, you could simply send a Linux/Unix "kill" command to signal for a heap dump*).

It's actually a good idea, since if the garbage is cleaned out, then the heap is a lot tidier for dumping purposes.

---
* "kill" is an unfortunate command name. It can send lots of different signals to a process and most of them are informational, not terminal.
 
Ben Evans
author
Posts: 67
10
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Tim,

Since the advent of JMX, heap dumps are triggered by calling a method exposed through JMX. It takes a boolean parameter, so it depends on whether the "perform GC" flag is set by the calling tool - as it happens, VisualVM always calls with it set to true.

You could call the exposed method with the parameter set to false - but you'd have to do so from a general JMX console, or through your own code.

In general, as you say, it's a good idea to minimise heap contents before performing a dump, as they are such expensive things to create.
 
Rancher
Posts: 4801
50
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
And, of course, in the situation where analysing a possible OOM via a memory leak you don't want to be worrying about stuff that would have been cleared out be a GC.  Anything a GC clears is not going to be part of the leak.
 
Tim Holloway
Saloon Keeper
Posts: 27762
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Ben Evans wrote:Hi Tim,

Since the advent of JMX, heap dumps are triggered by calling a method exposed through JMX. It takes a boolean parameter, so it depends on whether the "perform GC" flag is set by the calling tool - as it happens, VisualVM always calls with it set to true.



Actually, I'm pretty sure that when I was doing JMX as part of my job, it was Java 1.4. That particular snippet of my professional life ran from about 1.4 to 1.5. In fact, the JBoss engine of the time was JMX-based, from what I could see.

However, my employer at the time, like most in my town, was too cheap to spring for specialized debugging tools, so I had to make do with "kill" signals. Which, I would be very surprised to learn if they were no longer supported. They provide an alternative, if limited control mechanism for the JVM.

The main reason for not running GC before dumping the heap would probably be if you were on the JVM development team and needed to have a more complete look at JVM memory. Otherwise, a good pre-dump cleanup seems desirable.
 
Ben Evans
author
Posts: 67
10
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Tim,

Wow - we're really dredging up some ancient history here!

I could be wrong - but my recollection is that JMX first appears (at least in a network-accessible form) in Java 5 - not sure if it was available in a local form before that though.

The local tools, e.g. jstack, jmap and friends have (if memory serves) implementation-defined semantics on a local machine. On Linux, in the Java 8 era, they're actually implemented using Unix domain sockets, not signals. It may well be the case that a signals interface also exists for some of them - but I haven't tried.

However, as time passes, the command line tools are steadily being deprecated and removed - including hprof (which is signal based). I seem to recall that command line heap dump is also scheduled for removal, but can't remember with which version.
 
Tim Holloway
Saloon Keeper
Posts: 27762
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Even so. It was a long time ago, and I may be remembering that JMX was implemented as an external library prior to Java 5. Or I may just be hallucinating again.

Java is platform independent, but its original home was under Solaris (Unix), so the JVM implementations exploit Unix capabilities pretty heavily. Signals are still supported as follows: https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/signals006.html where some signals are only used internal to the JVM, and I believe kill -9 forces the OS to kill the process, external to the JVM.

There is no standard in the Java specs for any of this stuff, so what signals are honored and how depends on the vendor and version of the JVM in question.

The primary advantage of using signals in a Unix/Linux system is that they are a quick and simple way to alert the JVM without the overhead of special protocols and from anywhere in the OS, subject to OS security restrictions. Their disadvantage is that they are quick and simple, so you cannot qualify them with parameters (e.g., whether or not to run GC before a dump), and they are totally asynchronous by nature.

 
My first bit of advice is that if you are going to be a mime, you shouldn't talk. Even the tiny ad is nodding:
a bit of art, as a gift, that will fit in a stocking
https://gardener-gift.com
reply
    Bookmark Topic Watch Topic
  • New Topic