Win a copy of Microservices in Action this week in the Web Services forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Bear Bibeault
  • Devaka Cooray
  • Liutauras Vilda
  • Jeanne Boyarsky
Sheriffs:
  • Knute Snortum
  • Junilu Lacar
  • paul wheaton
Saloon Keepers:
  • Ganesh Patekar
  • Frits Walraven
  • Tim Moores
  • Ron McLeod
  • Carey Brown
Bartenders:
  • Stephan van Hulst
  • salvin francis
  • Tim Holloway

Use of Hibernate to insert million records from Flat File  RSS feed

 
Ranch Hand
Posts: 82
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi I am new to Hibernate and have secenario where I have to read a flat file parse it and insert into database. Now the flat file is pretty big that is may contain arround million records. It's taking too long to insert all the records if going through the traditional way so wondering if using hibernate how can I make it faster. But I am not that much familiar with hibernate so wondering if soomeone can help me out here I have used hibernate for simple stuff so far. Can someone provide a small sample or tutorial on this on reading a file and inserting the records into database. Any help is appreciated.

Thanks
 
ranger
Posts: 17314
11
IntelliJ IDE Mac Spring
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What do you mean by traditional way?

What database are you using, if you are using Oracle the SQLLoader is the fastest way to insert data from a flat file.

Mostly, bulk inserts are better off with straight JDBC than Hibernate, but you can do it, I doubt that it will give you faster speed than whatever "traditional" way you are doing.

Mark
 
Ranch Hand
Posts: 547
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
if you want to "load" the database with millions of records then you better do not go trough hibernate.
rather use the tools comming with your DB or (if you need to do it with java) use straight JDBC.
if, for some reason, you need to use hibernate: flush the session from time to time!
 
Mark Spritzler
ranger
Posts: 17314
11
IntelliJ IDE Mac Spring
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Originally posted by pascal betz:
if you want to "load" the database with millions of records then you better do not go trough hibernate.
rather use the tools comming with your DB or (if you need to do it with java) use straight JDBC.
if, for some reason, you need to use hibernate: flush the session from time to time!



That time to time should match the JDBC Batch size. So if that is 20 then flush every twenty records.

Mark
 
Don't get me started about those stupid light bulbs.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!