• Post Reply Bookmark Topic Watch Topic
  • New Topic

Out of memory work around for a java application (please help!)  RSS feed

 
Sameer Malhotra
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am working on a java application(Batch Process) and we have to use some third party API's (No Option to change it) related to the Database.It is causing out of memory after every 25000 updates and the total records are 5 million or so. I used the JProfiler to find out that the out of memory is caused by the third party API only.Since the application is in the production already we really cannot risk to change that third party API at any cost so we have to do some work around.

What is the best work around in this scenario? I am thinking to create separate thread in the below mentioned code which will do the first 20000 updates and as soon as this thread dies all the objects associated with this will also die and hence the new thread will be a fresh start and again the new thread will do 20000 updates or something like that.

If you have encountered such kind of problem in the past please help me and if possible write a pseudo code or sample code for it.

my current code looks like this which I am thinking to add the java threads:

public static void main(String [] args){

*ResultSet* rs (rs has all the records e.g. 5 millions)
long count=0;
while (rs.next()){

doUpdate();

count++;
*//the code fails if the count reaches to 25000*

}


}


Please do let me know what do you think my solution and if there is any better way please suggest.
 
Michael Kelly
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
If you want to do this in Java then the pattern would need to be something like

1) read n (where n is a fraction of the total and wont lead to OOME) rows from the source
2) write them to the destination DB
3) commit the transaction
4) repeat 1-3 until done.

However, when you're talking about loading five million rows using Java I think you're on the wrong track. All major DB vendors have loader tools for dealing with this kind of quantity.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!