Win a copy of Mastering Corda: Blockchain for Java Developers this week in the Cloud/Virtualization forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Paul Clapham
  • Ron McLeod
  • Bear Bibeault
  • Liutauras Vilda
Sheriffs:
  • Jeanne Boyarsky
  • Tim Cooke
  • Junilu Lacar
Saloon Keepers:
  • Tim Moores
  • Tim Holloway
  • Stephan van Hulst
  • Jj Roberts
  • Carey Brown
Bartenders:
  • salvin francis
  • Frits Walraven
  • Piet Souris

OutOfMemoryError when deviding to Bulks.

 
Ranch Hand
Posts: 294
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi All,
I have list of million of records,
I want to divide the list to smaller groups for later processing, say take 3000 keys each (Bulk) and insert to Bulk table.
Other process process these Bulks later on.

I based on code sample from Sun website (executeBatch tutorial) and made the following method

On 10000 list I get the Bulk dividing done in less then 2sec.
On 500000 List, the Bulk dividing take like 3 mints.
500000 and up I get out-of-memory Exception.

What do I do wrong?
Please advise.

Thanks
Sharon




 
author & internet detective
Posts: 40359
820
Eclipse IDE VI Editor Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Sharon,
Is the out of memory coming from the database part or the rest of the code? If you comment out the prepared statement part and don't get the error, it tells you that the problem is in the database code/driver. In that case, you can do commits more often so as not to build up as large a memory usage.
 
Sharon whipple
Ranch Hand
Posts: 294
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The out of memory coming from the Java side, problem was solved using -Xmx512m.

But the bulk splitting still take some time..

I added:

after each "addBatch" call so parameters will be removed.

And i added :


So the PreparedStatement will flush to the db every 100 records (6000 characters clob each)
But still this process takes large amount of time (11 seconds for just one million records) and memory space
What am I doing wrong?

Please advice.
Sharon
[ January 10, 2008: Message edited by: Sharon whipple ]
 
Jeanne Boyarsky
author & internet detective
Posts: 40359
820
Eclipse IDE VI Editor Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
11 seconds to insert one million records isn't bad. An import could take almost that long for a million records.

For the amount of memory used, you can check whether you need to hold all the data in memory at once. If you can do it in stages (read in some records, load them, repeat), this would save a lot of memory.
 
The two armies met. But instead of battle, they decided to eat some pie and contemplate this tiny ad:
the value of filler advertising in 2020
https://coderanch.com/t/730886/filler-advertising
reply
    Bookmark Topic Watch Topic
  • New Topic