Win a copy of Functional Reactive Programming this week in the Other Languages forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Use of Hibernate to insert million records from Flat File

 
Mike Boota
Ranch Hand
Posts: 82
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi I am new to Hibernate and have secenario where I have to read a flat file parse it and insert into database. Now the flat file is pretty big that is may contain arround million records. It's taking too long to insert all the records if going through the traditional way so wondering if using hibernate how can I make it faster. But I am not that much familiar with hibernate so wondering if soomeone can help me out here I have used hibernate for simple stuff so far. Can someone provide a small sample or tutorial on this on reading a file and inserting the records into database. Any help is appreciated.

Thanks
 
Mark Spritzler
ranger
Sheriff
Posts: 17278
6
IntelliJ IDE Mac Spring
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What do you mean by traditional way?

What database are you using, if you are using Oracle the SQLLoader is the fastest way to insert data from a flat file.

Mostly, bulk inserts are better off with straight JDBC than Hibernate, but you can do it, I doubt that it will give you faster speed than whatever "traditional" way you are doing.

Mark
 
pascal betz
Ranch Hand
Posts: 547
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
if you want to "load" the database with millions of records then you better do not go trough hibernate.
rather use the tools comming with your DB or (if you need to do it with java) use straight JDBC.
if, for some reason, you need to use hibernate: flush the session from time to time!
 
Mark Spritzler
ranger
Sheriff
Posts: 17278
6
IntelliJ IDE Mac Spring
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by pascal betz:
if you want to "load" the database with millions of records then you better do not go trough hibernate.
rather use the tools comming with your DB or (if you need to do it with java) use straight JDBC.
if, for some reason, you need to use hibernate: flush the session from time to time!


That time to time should match the JDBC Batch size. So if that is 20 then flush every twenty records.

Mark
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic