• Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

insert 10,00000 records using hibernate with struts2

 
Prakash Aradwad
Greenhorn
Posts: 15
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
i need to be insert 10,00000 records in table using hibernate at single time
for high performance

i am trying but not still working
 
Akhilesh Trivedi
Ranch Hand
Posts: 1608
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Prakash Aradwad wrote:i am trying but not still working


What did you try?
What did actually work or happen?
 
debraj mallick
Ranch Hand
Posts: 188
Hibernate Java
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
hi Prakash,

please give us the full stack trace, so that we could help.
 
Tim Moores
Bartender
Posts: 2955
46
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
If you want high performance, why are you using Hibernate? Have you looked into using ETL tools, or JDBC batch inserts?
 
Pramod Krishna Murthy
Ranch Hand
Posts: 37
Java Spring Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yes, I completely agree with the earlier post. even we did similar thing through batch updates. the performance is good and since it is plain java code, development exception handling, fall back are all easy to code
 
Prakash Aradwad
Greenhorn
Posts: 15
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
thanks friend's

ya i replaced hibernate with batchinsert using jdbc it works

through hibernate it is not possible to insert large data quickly.

 
leonardo ginting
Ranch Hand
Posts: 48
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Prakash Aradwad wrote:thanks friend's

ya i replaced hibernate with batchinsert using jdbc it works

through hibernate it is not possible to insert large data quickly.



hi prakashh by the way i have same issue with you but i couldnt figure it out why when i inserting 100000 rows it takes a long time?
how you can solve your problem?
 
Tim Moores
Bartender
Posts: 2955
46
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
A common reason is the presence of indexes on the table - every one of which will be updated for each insert. It's common practice to drop indexes before mass inserts, and re-add them later.

An ORM tool adds some overhead, and you normally don't need it for mass inserts. As I said before, that's the domain of ETL tools.
 
leonardo ginting
Ranch Hand
Posts: 48
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
hi Tim thanks for reply im new in this field anyway is there any other way i could speed up inserting mass rows using hibernate? i've been searching any tutorial for this and i still couldn figure it out by myself
 
Tim Moores
Bartender
Posts: 2955
46
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Just what has been suggested already: Use JDBC instead of Hibernate. Remove indexes before the insert. Make sure you don't use a separate transaction for each insert.
 
Kunal Lakhani
Ranch Hand
Posts: 622
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
A common reason is the presence of indexes on the table - every one of which will be updated for each insert. It's common practice to drop indexes before mass inserts, and re-add them later.

I didn't get that? What index are you all talking about?

Thanks
 
Tim Moores
Bartender
Posts: 2955
46
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
DB indexes on the table you're inserting into
 
Kunal Lakhani
Ranch Hand
Posts: 622
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Tim
 
leonardo ginting
Ranch Hand
Posts: 48
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Tim Moores wrote:Remove indexes before the insert.


hi tim thanks again for replying anyway i'm sorry i didn't get that? how could i remove indexes before insert? would you please explain to me how could i do that?
and how about
don't use a separate transaction for each insert

prior i used 1 EntityManager for many insert transaction each table, does it mean what you said about don't use a separate transaction for each insert?
 
Tim Moores
Bartender
Posts: 2955
46
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
leonardo ginting wrote:how could i remove indexes before insert? would you please explain to me how could i do that?

I'm talking about the SQL "DROP INDEX" statement. Check your DB's documentation for the exact syntax.

does it mean what you said about don't use a separate transaction for each insert?

Not sure what an EntityManager does by default; as I've said repeatedly, you should not use JPA or Hibernate or any other ORM for mass inserts. But I'm sure the documentation of EntityManager tells you how to start a transaction for multiple inserts, instead of using one for every single one.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic