Following 1.8 upgrade runtime fails

[Neil Barton]

I have a program that uses a lot of memory, I compile it within Netbeans using the flag -Xmx1500m in order to make a lot more available. When I run within Netbeans everything is fine the program runs. When I run the executable from a command line using java -jar executable.jar it fails "Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space".

It was working fine, the only thing I did was upgrade to the latest version of Java 1.8. The code is exactly the same and inside Netbeans it still works (even though there would be less memory because Netbeans is running!). When I go to Java control panel it says I am using Java V1.7, I need to keep that on there to access a 32 bit ODBC driver but I did expect to see V1.8 as well. If I run java -version from my command line I get 1.8.0_65. Why does it work in Netbeans but not in Live?

[Simon Roberts]

Have you tuned the GC in your command line? If so, the tuning parameters might be interpreted differently in Java 8 (particularly if it decided to default to G1GC instead of CMS). But, if not, nothing else springs to mind immediately as to why it would fail suddenly. Others are likely smarter than I however.

If the program stays up for a while, you could start it and connect jvisualvm (standard tool in the JDK) and poke around to see what the memory is doing.

If it dies promptly, you'll perhaps benefit from forcing a heap dump on out of memory, then analyzing the resulting heap dump:

  java -XX:+HeapDumpOnOutOfMemoryError my.app.Main

Though if it dies promptly, that might simply be different defaults, so maybe just give it a bigger heapsize. But I presume you tried that first ...

[Neil Barton]

OK, after some experimentation I've come to the conclusion that putting the -Xmx1500m flag into Netbeans run time box doesn't work. Putting it on the command line does. so when I run the command:
java -Xmx1000m exec.jar it works.
A shame I can't keep it all in the development env.

[Neil Barton]

OK, I got it to run with the command java -Xmx1000m -jar exec.jar but it is so slow it will never finish. When it used to run it took up all of the CPU and just about all of the memory but now it barely touches it so clearly this command is not allocating the memory either. So my question is: with version Java 1.8.65 how do I allocate heap space?

[Simon Roberts]

Well, it should be the same option at that level. It's the -XX: options that are unstable between releases.

I'm presume you are sure you're specifying this as a VM option and not as a command line option?

It works for me, not that this helps you much...

[Neil Barton]

I put some debug in the top of the program. Here is the output when running in Netbeans where the program works fine:
Total Memory (in bytes): 268435456
Free Memory (in bytes): 266884192
Max Memory (in bytes): 1864368128
... and this is run from the command line where I get the outofmemory error:
Total Memory (in bytes): 16252928
Free Memory (in bytes): 13696120
Max Memory (in bytes): 259522560

In Netbeans in the properties of the project I have an entry in the VM options: -Xmx2000m
I don't understand :-(
Why are total and free memory greater when running inside Netbeans (given that it should be less as Netbeans is running!) than when running on the command line?
Why did it work before I upgraded to V65?
How do I tell the VM to use more memory given that the flag in Netbeans seems not to work, nor does it work on the command line?

[Simon Roberts]

Well, according to that, you have plenty more heap available, and you are getting a bigger max memory in the failing configuration.

Printing these things at startup isn't all that interesting, beyond the limit. How long does this run for before it dies? It would be more interesting to run it and watch its behavior under jvisualvm. if you can't do that, then you must generate a heap dump and do post mortem analysis on that.

Frankly, I would have thought it was running out of PermGen, except that went away in Java 8. Anyway, without more data, you're just guessing and that's pointless.

I suggest you configure it to generate a heapdump on out of memory error, and then look at that, probably in jvisualvm anyway. At least that way you'll be able to look at the contents, and quantity, of memory at the point of failure, and compare it with the non-failing version to see what's happening.

[Neil Barton]

Okay, maybe I need to explain more of what I'm doing...

My program creates and object called a chromosome with lots of attributes (alleles). These objects are held in an arraylist. I create the arraylist at the start so I can go through and analyse each one to see how it performs, then I rank them and put the best ones at the top.

the program has been running for about 5 years with a population of 5000 in the arraylist. Since the version 1.8 upgrade it creates 3796 and then fails with the error: Exception in thread "AWT-EventQueue-0", java.lang.OutOfMemoryError: Java heap space.

On the line that it fails on I am adding an item into the arraylist (the 3797th).

If I reduce the population in the arraylist to 3500 it runs out of memory in another part of the program.

I have -Xmx1500m in the VM part of netbeans and have tried various flags (-XMS100m -Xmx1500m) on the command line. None of them seem to make the slightest difference to the running of the program. How do I increase the amount of heap space in V1.8? The switches that worked in 1.7 no longer seem to work.

[Neil Barton]

I uninstalled Java 1.8. Downloaded and re-installed it. It works.
I don't understand :-(
I'm just happy it's working again :-)

[Simon Roberts]

What JVM are you using? in the Oracle JVM, those flags are still defined. They are, in theory "non-standard" and not guaranteed to be supported by every VM, but I've never heard of one that doesn't support -Xmx and -Xms. Also, your output already showed that it's working, so I believe something else is wrong.

I suggest you add the heap dump on out of memory error command and take a post-mortem look at the program in jvisualvm to see if you can see what's been allocated. There's a tiny, but non-zero, chance that you've found a bug in the JVM, I suppose.

Oh, one other thought. Did you move from a 32 bit JVM to a 64 bit one at the same time as you made the upgrade?

And while not promising anything, are you willing / able to share your program, so others can try running it and see how it behaves (would it even be possible for non biologists to make it work to the point of failure?

[Neil Barton]

Thanks Simon,

I appreciate your interest but I think I've accepted that it was something that screwed up in the installation of 1.8. whatever it did I don't think it can be reproduced!

This particular program has, I think, reached the limit of what it can do (sorting a mass of data using a genetic algorithm to get the best set of parameters to rise to the top). It has shown that it's not completely useless and I now have another theme to follow that will exploit this approach, as ever, one piece of code just exposes another problem to solve :-) As such I don't think it's helpful for me to share my code at the moment as I've accepted that it's a blind alley. However, when I get it working in a more streamlined way I may publicise what I've been doing in a way which other people can experiment with to try and solve other problems.

Thanks for your help, people working alone like me need forums like this to help us take a step forward (or sideways) when it looks like we're banged right up against the wall. You telling me that it was working for you meant it did work and I just had to find out why mine didn't.

Thanks again,
Neil

[Simon Roberts]

Fair enough. Good luck with it. I'd be happy to try it on my installation if you feel like--weird memory behavior holds a bit of fascination for me. Of course, I would have to be able to install and run it without understanding the particular problem domain

Cheers
Simon