java.lang.OutOfMemory error

[samit majumdar]

My java program parses an xml file, collects data from there and inserts data to rdbms or it can even display the data in tabular format in a browser. I have written two versions of my program... one using DOM and other one is using JDOM. Both are working fine for handling 15,000 to 20,000 of records. But in reality my program have to handle millions of records... may be 15 to 20 million..or you may consider the amount as unlimited. Here comes the problem... I am getting java.lang.OutOfMemory error. How to handle that? If anybody has any solution, please give it to me as soon as possible...i am in urgent need of it...3

Thanks-
Samit Kumar Majumdar

[Paul Sturrock]

You might want to consider processing this with a SAX parser, rather then using DOM. SAX is event driven, so you define a handler which reacts to the start of an XML element and does some things accordingly. This way you should be able to handle this enormous document bit by bit. The design of DOM requires the whole document be loaded before anything can be done with it.

[Jeroen Wenting]

yes, for something this big DOM is simply not suitable (especially as you state the amount of data can be considered infinite).

SAX it will have to be, or else write your own system that breaks the stream into managable chunks and create an XML document from each chunk which you then can parse using a DOM parser.
That of course will likely be a lot more work than using a SAX parser (and may well require a SAX parser internally).

[samit majumdar]

HI!,
I tried it out with pure SAX....but still it is giving java.lang.OutOfMemoryError. Do u have alternate ways...if so...please give it ...I am extremely in need of it.
Regards-
Samit

[sandip mense]

Samit,

You can increase the memory of the jvm and try, When you start the the process by giving java -Xmx300m javaFile
If you are running the application in a application server, checkout the documentation of the server for increasing the size of jvm. IBm websphere provides the means through the GUI for every instance you create.

Sandip

[Liz Burke-Scovill]

Can you page through it and take x number of records at a time, process them, and then move forward?

We had a similar situation with a different API and technology with the Domino API in trying to parse through a view and reformat to return to the browser. Anytime we got over n number of records, the JVM started throwing Out of Memory errors, and other errors reminiscent of C++ errors.

In addition to paring down whatever objects we had (Notes objects are inherently memory hogs) to non-Notes objects, and tweaking code so that it behaved more dynamically, we also ended up adding filters to pare down the number of records, and then essentially paging it so that it was done chunks at a time.

Eventually we managed to get it to a point where we could handle all records without memory errors. However, we never had a situation where we were processing on the order of millions. Mostly 100's of thousands.

[Stan James]

I'm with the suggestion for SAX. If you're just reading XML and writing to a database you should be able to hold down the amount of stuff in memory at one time. For the HTML output, are you building it in memory or writing it to the HTTP response stream? Building it all in memory could easily lead to "out of memory" problems.

[Josh Allen]

Originally posted by samit majumdar:

I tried it out with pure SAX....but still it is giving java.lang.OutOfMemoryError.

Some SAX parsers don't start parsing (raising events) until EOF. Check to make sure this isn't the case with the one you are using. I did hack a SAX parser once because I was using it to parse a custom protocol from a socket (similar to Jabber) and it hung waiting for EOF. You should still be able to find it under phosphor.sf.net source.

[samit majumdar]

Hi Everybody!,
Thanks for your guidance...I solved the problem by storing the data in multiple xml files(each containing 50,000 records) and viewing or restoring from those files...definitely, by using SAX.
thanks again,
Regards-
SAMIT