This week's book giveaway is in the Agile and Other Processes forum.
We're giving away four copies of The Journey To Enterprise Agility and have Daryl Kulak & Hong Li on-line!
See this thread for details.
Win a copy of The Journey To Enterprise Agility this week in the Agile and Other Processes forum! And see the welcome thread for 20% off.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Jeanne Boyarsky
  • Liutauras Vilda
  • Campbell Ritchie
  • Tim Cooke
  • Bear Bibeault
Sheriffs:
  • Paul Clapham
  • Junilu Lacar
  • Knute Snortum
Saloon Keepers:
  • Ron McLeod
  • Ganesh Patekar
  • Tim Moores
  • Pete Letkeman
  • Stephan van Hulst
Bartenders:
  • Carey Brown
  • Tim Holloway
  • Joe Ess

To read and save high volume of data.  RSS feed

 
Ranch Hand
Posts: 455
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
hi,

I want to read a 5 GB file which may contain 6 millions records (may be less) and i need to save this records to database.

What is the best approach to do this ? If i read the data from the file and store it in memory before saving to database it sure will take lots of memory ! I appretiate any ideas or thoughts on this.

Thanks,
Rajeev.
 
Ranch Hand
Posts: 1780
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Must you use Java?
 
Rajeev Ravindran
Ranch Hand
Posts: 455
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Jeff: Thanks for the instant reply..

No, Java is not mandatory . If not java, what you think is the other option ?

Thanks,
 
Jeff Albertson
Ranch Hand
Posts: 1780
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
There is a whole industry in such db tools. I'm just a lowly Java coder and when all you have is a hammer everthing starts to looks like nails If you just need to do this once or infrequently then it probably doesn't matter how you do this. In Java, you would use a PreparedStatement and its batch methods to batch your inserts, if you can.
 
Rajeev Ravindran
Ranch Hand
Posts: 455
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
This process is not that frequent, may be i have to run it once in a quarter !

Any other suggestions ? Is there any better approach than using java ?

Thanks,
 
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Jeff smoked the answer for ya -

most real databases have tools that provide functionality to import with ETL, so if you are using SQL Server, Oracle, DB2 (egads i just choked on bile listing DB2 as a real database) you can write scripts to import your data from say a delimited text file to database tables. you can even schedule them.
 
Rajeev Ravindran
Ranch Hand
Posts: 455
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks jeff / Greg. We are using AS400 , may be we will go ahead with some RPG program to read data from file. Thanks againg for your inputs !

Jeff, watz ur SCJP score ? just curious

Thanks,
 
Jeff Albertson
Ranch Hand
Posts: 1780
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Originally posted by Rajeev Ravindran:
Jeff, watz ur SCJP score ? just curious



Clue: what's the lowest score for a pass?
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!