• Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Large no. of Records problem

 
sandeep kokane
Greenhorn
Posts: 19
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi All,
We are develping an applicaion which moves data from one database to another. We are using jsp,jdbc and tomcat. But we are facing problem when we are moving large data (around 5 Laks records) Our system crashes when we tried to do this.
How to handel this issue. Also i want to know how we can increase our preformance regarding this issue. And how we can handle the transaction issues.
Thanks in Advance,
Sandeep
 
Peter den Haan
author
Ranch Hand
Posts: 3252
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Unfortunately you don't give any information on the crash (A Java exception? If so, what's the stack trace? An OS exception? If so, what type and is it the database or the application server? A hang? Database or application server?)
Also please give some more technical detail. 500,000 records - are you doing all of it in one transaction? Are you maybe running out of space in the database because of this? What database and JDBC driver are you using? How many records can you transfer without crash?
Generally, for best performance use the JDBC batch facility to do the inserts, and commit regularly if you don't desperately need the transactional safety. Maybe start with 100 batched updates at a time, a commit every 1000 updates, and take it from there - try varying the numbers to see what performance impact it has.
- Peter
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic