• Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Improving performance of Hibernate based batch program

 
Richard Ellison
Ranch Hand
Posts: 51
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello,
We have a batch program which moves data from one database to another.But it takes an unusually long time to convert the data- almost 2 hours to move 40000 records.Would connection pooling help optimize the program?Please advise.

Thanks
Gary
 
Paul Sturrock
Bartender
Posts: 10336
Eclipse IDE Hibernate Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Perhaps. My guess though is its memory, extra processing load and probably poor SQL that are your bottlenecks . Have you done any timings for the various stages in the process? If your program waiting for the database or the database waiting for your program?

One thing worth noting, Hibernate (like any ORM) is not a great fit for bulk operations like this - it's not really their primary purpose. Usually, if performance is your key concern, the vendor specific bulk loading tools are much better.
 
Richard Ellison
Ranch Hand
Posts: 51
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Paul for your quick response.How long do you think it should take for a batch program to move data from one database to another-around 40 tables involved and around 40,000 records being moved.

Thanks again.

Gary
 
David Newton
Author
Rancher
Posts: 12617
IntelliJ IDE Ruby
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yeah, this is not something I would do with Hibernate, *particularly* if it's the exact same table structure. (And even if it isn't, I can't see Hibernate being the fastest solution.)
 
Paul Sturrock
Bartender
Posts: 10336
Eclipse IDE Hibernate Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Richard Ellison wrote:Thanks Paul for your quick response.How long do you think it should take for a batch program to move data from one database to another-around 40 tables involved and around 40,000 records being moved.

Thanks again.

Gary


How long is a piece of string?

It depends on the database, the hardware, the data model, the type of data, the network etc. Where database tools do this well is they can control these factors more than a generic tool like an ORM can. Oracle's bulk loader for example can selectively turn of constraints and indexes, thread the load out into multiple concurrent jobs and insert data in chunks that best use the stroage configuration of the underlying database. If performance is a real squeeze I would only consider your own database's bulk loading tool.
 
Richard Ellison
Ranch Hand
Posts: 51
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks to all for the tips.I guess at this point I have to try to optimize performance this Hibernate-based program since it is too late for me to go back and redevelop.

Thanks once again.

Richard
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic