Win a copy of Escape Velocity: Better Metrics for Agile Teams this week in the Agile and Other Processes forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Liutauras Vilda
  • Tim Cooke
  • Paul Clapham
  • Jeanne Boyarsky
Sheriffs:
  • Ron McLeod
  • Frank Carver
  • Junilu Lacar
Saloon Keepers:
  • Stephan van Hulst
  • Tim Moores
  • Tim Holloway
  • Al Hobbs
  • Carey Brown
Bartenders:
  • Piet Souris
  • Frits Walraven
  • fred rosenberger

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

 
Ranch Hand
Posts: 428
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I am writing a program to dump a sql script will all the data from a database. I am doing this rather than using the Oracle Export because some data is sensitve and needs to be scrubbed out.

The database is big, and I am getting this memory error. I do have several buffers, and I am cleaning them out to manage the space. I am also doing a flush after each write.

I was writing the commands out in chunks, but it died really fast. Now with individual writes, it takes longer, but I still get this memory error.
 
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You need to increase the heap space allocated to Java. This is a parameter on the JVM startup:

-Xmx1024m will give 1024 megabytes to the JVM so it will take that as needed
-Xms512m will make the heap initially 512 megabytes, without resizing up from the default. This will save time that would be spent in heap resizing.

As long as your program is not in an infinite loop, and the computer can handle whatever you're reading in, resizing the heap to something appropriately larger should fix your problem.
 
M Burke
Ranch Hand
Posts: 428
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks Kai, that worked.

I just wonder why it works. I flushed the buffer after every write, yet it still died. I even tried closing the file and reopeniong it in append mode afer every 5000 writes, and it still died in the same place.
 
author
Posts: 14112
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by M Burke:
Thanks Kai, that worked.

I just wonder why it works. I flushed the buffer after every write, yet it still died. I even tried closing the file and reopeniong it in append mode afer every 5000 writes, and it still died in the same place.



The answer is, of course, that it's not the stream that's taking up the memory, but something else.
 
Kai Johnson
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by M Burke:
I just wonder why it works. I flushed the buffer after every write, yet it still died. I even tried closing the file and reopeniong it in append mode afer every 5000 writes, and it still died in the same place.




Memory profiling is usually an enlightening experience. I suggest using a profiler to find out what was really using up all the memory. In my experience, it's the silly mistakes, like not pre-sizing StringBuilders and StringBuffers, or accidentally holding onto objects in collections so they aren't garbage collected.

JProfiler has a free trial license, try it out:
JProfile trial
 
Greenhorn
Posts: 16
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
There is explanation of program not working on small heap size , before going ahead I would tell that you are lucky that you did not get the same error after increasing the heap size to 1024MB, if the records were larger you still can get the same error.

Here howz things work, I would not go into the detail of GC and young zen,old zen but the problem is not free space rather contiguos free space.

My explanation:
What GC does: mark,sweep and compact, in this process there are different algorithm for different JDK provider like sun,ibm and Bea systems.

Here is an example lets say total memory available is 100K
you need 10K for your single operation, it might be possible that even if 80K memory is available you can get outofmemory exception, lets assume there are some occupied memory at every 8K then the program will not find the contiguos 10K memory anywhere in the Heap and it will give out of memory exception.
I know I am not clear here ..but I can explain it better with visual diagram
 
I was her plaything! And so was this tiny ad:
the value of filler advertising in 2021
https://coderanch.com/t/730886/filler-advertising
reply
    Bookmark Topic Watch Topic
  • New Topic