• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Ron McLeod
  • Rob Spoor
  • Tim Cooke
  • Junilu Lacar
Sheriffs:
  • Henry Wong
  • Liutauras Vilda
  • Jeanne Boyarsky
Saloon Keepers:
  • Jesse Silverman
  • Tim Holloway
  • Stephan van Hulst
  • Tim Moores
  • Carey Brown
Bartenders:
  • Al Hobbs
  • Mikalai Zaikin
  • Piet Souris

Handling massive number of records

 
clojure forum advocate
Posts: 3479
Mac Objective C Clojure
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hey,
In our application we have read one day record from MySQL database, make a little change on each retrieved record and then save the records to a new database.
Please note that each day holds a massive number of records (could be a million records).
What is the best strategy to implement this?
I'm thinking to use MySQL limit keyword to read each 100 row, do you have any better approach?
And what about saving the retrieved/modified each 100 row?
Using a batch?
Thanks.
 
author
Posts: 4278
34
jQuery Eclipse IDE Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
If it's in millions of records I hope you have the two databases physically sitting near each other or network latency is going to be high. For such a situation though, you may consider performing a backup up the database into a single file, transferring the file to the target database machine, then unloading/converting it there.

It seems like a lot of work but databases are known to thrash for high number of records, and getting JDBC to do batch records optimally can be hard.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
reply
    Bookmark Topic Watch Topic
  • New Topic