OutOfMemoryError killing me

[Sam Smith]

Hi,

I'm trying to run a genetic algorithms experiment, and keep getting a java.lang.OutOfMemoryError error.

This happens when I'm trying to populate my world to set up the experiment.

I'm trying to create 50 environments, each of which contain an array with 1000 organisms, each of which contain an array with 10 neurons, each of which contain (on average) an array of 3 random weights (floats).

I'm therefore basically trying to create 50*1000*10*3 = 1,500,000 floats.

The program crashes after creating about 25 environments, so 750,000 floats. At smaller numbers the program doesn't crash.


So a couple of questions: 750,000 floats doesn't sound like a huge amount for a program to create. Is this enough to get an OutOfMemory error?

Is it likely that my datasets are inefficient? Arrays should be pretty light-weight, right? I initialize them once only, with the correct size.

To be able to do further testing, is there any way to back-trace the OutOfMemory error? I'm using eclipse, and, while you can click on many errors to see where they're coming from, I can't with this one.


Any advice or recommendations would be much appreciated.

Thanks,
Sam

[Jeffrey Spaulding]

Did you increase the memory footprint of Eclise?

Run eclipse.exe -vmargs -Xmx256m (or whatever your machine can give you)

Does it work when running outside of eclipse

If not start your vm with a bigger heap

http://java.sun.com/j2se/1.3/docs/tooldocs/solaris/java.html

[Sam Smith]

Originally posted by Jeffrey Spaulding:
Did you increase the memory footprint of Eclise?

Run eclipse.exe -vmargs -Xmx256m (or whatever your machine can give you)

Does it work when running outside of eclipse

If not start your vm with a bigger heap

http://java.sun.com/j2se/1.3/docs/tooldocs/solaris/java.html

Hmmm, it doesn't work outside of Eclipse, though the error message is a little more detailed:

'Exception in thread "Thread-2" java.lang.OutOfMemoryError: Java heap space'

I'm not certain where in the link you gave me it has an option for increasing the heap, so I haven't been able to test that yet. Also, is it not possible to request more space from inside the code?

Thanks

[Sam Smith]

Oh wait java -Xmx256m works.

So is there any way to set things so I don't have to use this each time?

And if I increase this further will the program run any faster (it's really slow at such numbers)? Is there a point that I shouldn't exceed for fear of breaking the computer?
[ June 10, 2005: Message edited by: Sam Smith ]

[Joe Ess]

Originally posted by Sam Smith:
Oh wait java -Xmx256m works.
So is there any way to set things so I don't have to use this each time?

No.

And if I increase this further will the program run any faster (it's really slow at such numbers)?

No. The size of the heap will not make your program run faster. Large heap sizes will make program startup slightly slower since the VM must allocate and configure more memory. If memory requirements of running programs exceeds physical memory, your computer will swap memory to disk "virtual memory". This situation can cause serious performance problems as the CPU has to manage swapping and can't spend valuable time running programs.
Slow performance in your case may have something to do with your program managing 1,500,000 independent organisms.

Is there a point that I shouldn't exceed for fear of breaking the computer?

Physically breaking it? No.
Bringing it to a crawl and possibly crashing it? Yes. As above, total up the memory requirements of all running processes and don't let that number exceed physical memory.

[Sam Smith]

Originally posted by Joe Ess:
Slow performance in your case may have something to do with your program managing 1,500,000 independent organisms.

Well, that's unavoidable.

Physically breaking it? No.
Bringing it to a crawl and possibly crashing it? Yes. As above, total up the memory requirements of all running processes and don't let that number exceed physical memory.

Well yes, I meant break it metaphorically - crash it. But, with an otherwise idle Windows XP machine (except for backgroup processes), 512 MB RAM, 7.2 GB free space on my hard drive, it shouldn't complain too much if I, say, use -Xmx512m, would it? Just wondering in case I start needed to run even bigger experiments


... but anyway, I don't think it would come to that. Thanks very much for your help.
[ June 10, 2005: Message edited by: Sam Smith ]

[Joe Ess]

Originally posted by Sam Smith:


But, with an otherwise idle Windows XP machine (except for backgroup processes), 512 MB RAM, 7.2 GB free space on my hard drive, it shouldn't complain too much if I, say, use -Xmx512m, would it?

[ June 10, 2005: Message edited by: Sam Smith ]

Your computer should not complain initially because -Xmx sets the maximum heap size. The VM is going to be much smaller than that at startup (default is 2mb, you should probably use -Xms to set a reasonable starting size). Depending on the overhead of XP, trying to fill RAM with your app could cause some serious swapping and all the unstability that comes along with it. If you're really concerned, keep an eye on your computer's resources and increment your test populations in reasonable numbers.