• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Jeanne Boyarsky
  • Liutauras Vilda
  • Campbell Ritchie
  • Tim Cooke
  • Bear Bibeault
Sheriffs:
  • Paul Clapham
  • Junilu Lacar
  • Knute Snortum
Saloon Keepers:
  • Ron McLeod
  • Ganesh Patekar
  • Tim Moores
  • Pete Letkeman
  • Stephan van Hulst
Bartenders:
  • Carey Brown
  • Tim Holloway
  • Joe Ess

Use of Hibernate to insert million records from Flat File  RSS feed

 
Ranch Hand
Posts: 82
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi I am new to Hibernate and have secenario where I have to read a flat file parse it and insert into database. Now the flat file is pretty big that is may contain arround million records. It's taking too long to insert all the records if going through the traditional way so wondering if using hibernate how can I make it faster. But I am not that much familiar with hibernate so wondering if soomeone can help me out here I have used hibernate for simple stuff so far. Can someone provide a small sample or tutorial on this on reading a file and inserting the records into database. Any help is appreciated.

Thanks
 
ranger
Sheriff
Posts: 17314
11
IntelliJ IDE Mac Spring
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What do you mean by traditional way?

What database are you using, if you are using Oracle the SQLLoader is the fastest way to insert data from a flat file.

Mostly, bulk inserts are better off with straight JDBC than Hibernate, but you can do it, I doubt that it will give you faster speed than whatever "traditional" way you are doing.

Mark
 
Ranch Hand
Posts: 547
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
if you want to "load" the database with millions of records then you better do not go trough hibernate.
rather use the tools comming with your DB or (if you need to do it with java) use straight JDBC.
if, for some reason, you need to use hibernate: flush the session from time to time!
 
Mark Spritzler
ranger
Sheriff
Posts: 17314
11
IntelliJ IDE Mac Spring
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Originally posted by pascal betz:
if you want to "load" the database with millions of records then you better do not go trough hibernate.
rather use the tools comming with your DB or (if you need to do it with java) use straight JDBC.
if, for some reason, you need to use hibernate: flush the session from time to time!



That time to time should match the JDBC Batch size. So if that is 20 then flush every twenty records.

Mark
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!