Win a copy of React Cookbook: Recipes for Mastering the React Framework this week in the HTML Pages with CSS and JavaScript forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Ron McLeod
  • Paul Clapham
  • Rob Spoor
  • Liutauras Vilda
Sheriffs:
  • Jeanne Boyarsky
  • Junilu Lacar
  • Tim Cooke
Saloon Keepers:
  • Tim Holloway
  • Piet Souris
  • Stephan van Hulst
  • Tim Moores
  • Carey Brown
Bartenders:
  • Frits Walraven
  • Himai Minh

To read and save high volume of data.

 
Ranch Hand
Posts: 455
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
hi,

I want to read a 5 GB file which may contain 6 millions records (may be less) and i need to save this records to database.

What is the best approach to do this ? If i read the data from the file and store it in memory before saving to database it sure will take lots of memory ! I appretiate any ideas or thoughts on this.

Thanks,
Rajeev.
 
Ranch Hand
Posts: 1780
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Must you use Java?
 
Rajeev Ravindran
Ranch Hand
Posts: 455
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Jeff: Thanks for the instant reply..

No, Java is not mandatory . If not java, what you think is the other option ?

Thanks,
 
Jeff Albertson
Ranch Hand
Posts: 1780
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
There is a whole industry in such db tools. I'm just a lowly Java coder and when all you have is a hammer everthing starts to looks like nails If you just need to do this once or infrequently then it probably doesn't matter how you do this. In Java, you would use a PreparedStatement and its batch methods to batch your inserts, if you can.
 
Rajeev Ravindran
Ranch Hand
Posts: 455
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
This process is not that frequent, may be i have to run it once in a quarter !

Any other suggestions ? Is there any better approach than using java ?

Thanks,
 
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Jeff smoked the answer for ya -

most real databases have tools that provide functionality to import with ETL, so if you are using SQL Server, Oracle, DB2 (egads i just choked on bile listing DB2 as a real database) you can write scripts to import your data from say a delimited text file to database tables. you can even schedule them.
 
Rajeev Ravindran
Ranch Hand
Posts: 455
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks jeff / Greg. We are using AS400 , may be we will go ahead with some RPG program to read data from file. Thanks againg for your inputs !

Jeff, watz ur SCJP score ? just curious

Thanks,
 
Jeff Albertson
Ranch Hand
Posts: 1780
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by Rajeev Ravindran:
Jeff, watz ur SCJP score ? just curious



Clue: what's the lowest score for a pass?
 
"Don't believe every tiny ad you see on the internet. But this one is rock solid." - George Washington
the value of filler advertising in 2021
https://coderanch.com/t/730886/filler-advertising
reply
    Bookmark Topic Watch Topic
  • New Topic