• Post Reply Bookmark Topic Watch Topic
  • New Topic

To read and save high volume of data.  RSS feed

 
Rajeev Ravindran
Ranch Hand
Posts: 455
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
hi,

I want to read a 5 GB file which may contain 6 millions records (may be less) and i need to save this records to database.

What is the best approach to do this ? If i read the data from the file and store it in memory before saving to database it sure will take lots of memory ! I appretiate any ideas or thoughts on this.

Thanks,
Rajeev.
 
Jeff Albertson
Ranch Hand
Posts: 1780
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Must you use Java?
 
Rajeev Ravindran
Ranch Hand
Posts: 455
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Jeff: Thanks for the instant reply..

No, Java is not mandatory . If not java, what you think is the other option ?

Thanks,
 
Jeff Albertson
Ranch Hand
Posts: 1780
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
There is a whole industry in such db tools. I'm just a lowly Java coder and when all you have is a hammer everthing starts to looks like nails If you just need to do this once or infrequently then it probably doesn't matter how you do this. In Java, you would use a PreparedStatement and its batch methods to batch your inserts, if you can.
 
Rajeev Ravindran
Ranch Hand
Posts: 455
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
This process is not that frequent, may be i have to run it once in a quarter !

Any other suggestions ? Is there any better approach than using java ?

Thanks,
 
Greg Fidster
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Jeff smoked the answer for ya -

most real databases have tools that provide functionality to import with ETL, so if you are using SQL Server, Oracle, DB2 (egads i just choked on bile listing DB2 as a real database) you can write scripts to import your data from say a delimited text file to database tables. you can even schedule them.
 
Rajeev Ravindran
Ranch Hand
Posts: 455
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks jeff / Greg. We are using AS400 , may be we will go ahead with some RPG program to read data from file. Thanks againg for your inputs !

Jeff, watz ur SCJP score ? just curious

Thanks,
 
Jeff Albertson
Ranch Hand
Posts: 1780
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Rajeev Ravindran:
Jeff, watz ur SCJP score ? just curious


Clue: what's the lowest score for a pass?
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!