• Post Reply Bookmark Topic Watch Topic
  • New Topic

Use of Hibernate to insert million records from Flat File  RSS feed

 
Mike Boota
Ranch Hand
Posts: 82
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi I am new to Hibernate and have secenario where I have to read a flat file parse it and insert into database. Now the flat file is pretty big that is may contain arround million records. It's taking too long to insert all the records if going through the traditional way so wondering if using hibernate how can I make it faster. But I am not that much familiar with hibernate so wondering if soomeone can help me out here I have used hibernate for simple stuff so far. Can someone provide a small sample or tutorial on this on reading a file and inserting the records into database. Any help is appreciated.

Thanks
 
Mark Spritzler
ranger
Sheriff
Posts: 17309
11
IntelliJ IDE Mac Spring
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What do you mean by traditional way?

What database are you using, if you are using Oracle the SQLLoader is the fastest way to insert data from a flat file.

Mostly, bulk inserts are better off with straight JDBC than Hibernate, but you can do it, I doubt that it will give you faster speed than whatever "traditional" way you are doing.

Mark
 
pascal betz
Ranch Hand
Posts: 547
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
if you want to "load" the database with millions of records then you better do not go trough hibernate.
rather use the tools comming with your DB or (if you need to do it with java) use straight JDBC.
if, for some reason, you need to use hibernate: flush the session from time to time!
 
Mark Spritzler
ranger
Sheriff
Posts: 17309
11
IntelliJ IDE Mac Spring
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by pascal betz:
if you want to "load" the database with millions of records then you better do not go trough hibernate.
rather use the tools comming with your DB or (if you need to do it with java) use straight JDBC.
if, for some reason, you need to use hibernate: flush the session from time to time!


That time to time should match the JDBC Batch size. So if that is 20 then flush every twenty records.

Mark
 
Don't get me started about those stupid light bulbs.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!