the joy of Out Of Memory

[prateek urmaliya]

Hello Brethren,

I am creating a Batch Tool that can be run overnight and generates report on HomoSapiens on earth,
All it does is to pull some data from DB based on few constraints provided in an XLS let�s say FooNos. And BarNos.,
Than process this info and generate MereMortals that will further be mutated
Into HomoSapiens based on certain features,
And finally this collection will end up in xls format with all the bells and whistles,
I am using Jexcel for both reading the selection criterion and generating the report.
now the proverbial rubber hits the road once I get the RS and process it into a collection of MereMortals and then use this collection for further manipulation that again generates some more HomoSapiens,The no. of HomoSapiens generated occupies the Heap n keep growing,
I pulled my JMP from attic and figured out that the huge no. of rows that I am getting is resulting in this many object, I spruced code a bit to generate lesser no. of object,
I also put constraint (by limiting FooNos. And BarNos) on the no. of MereMortals to be processed at any given time,
But than came the reporting part now to write all these HomoSapiens in XLS needs some more memory and the batch crashes,
The no. of MereMortal that I am processing is over 150K,
so I am breaking it so not more than 30K can be there on heap at any given time but to write these MereMortals in XLS again will take more memory,
I am aiming at generating single XLS with multiple sheets each one will be added for
Next lot of HomoSapiens, I aim at keeping minimal no of sheet,
I am running a reporting tool while the tool queries for the next lot,
Will using CSV as pipe will help and then reassembling the entire sheets (CSV) in XLS.
What I can do differently to save the show
[ January 05, 2007: Message edited by: prateek urmaliya ]

[Joe Ess]

You have two options: Make the VM larger or make the problem smaller. You can give the VM more memory through the -Xmx command-line switch described here. You'll also probably have to massage your logic to conserve memory, like reading the result set directly into a temp file rather than storing it in-memory as a collection.

[prateek urmaliya]

VM is already occupying Luxury Real Estate,
Well I have done some changes to reduce memory footprint but still I do need to Iterate through MereMortals to get HomoSapiens.
What else I can do Will CSV help?
[ January 05, 2007: Message edited by: prateek urmaliya ]

[Joe Ess]

Assuming there's no relationship between HomoSapiens, there's no good reason to load thousands of them into memory. You could pull 1 (or 10 or 1024) out of the database, process them and go back for more.
Using CSV's may make sense if Jexcel insists on having the entire XLS document in-memory. Of course, if you expect to assemble the CSV's into an XLS using Jexcel, you'll probably have the same problem (assuming that it wants to represent the whole document in-memory).
Can you get away with having a CSV be the end product?

[prateek urmaliya]

Thanks Joe,

well I reduced the problem space,
now I am processing a fix no. of mere mortals at a given time,
and generating report for this batch of homosapiens,
than provide hyperlink of all these reports in a summary report with all the formating info (bells and whistles).