• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Bear Bibeault
  • Paul Clapham
  • Jeanne Boyarsky
  • Knute Snortum
Sheriffs:
  • Liutauras Vilda
  • Tim Cooke
  • Junilu Lacar
Saloon Keepers:
  • Ron McLeod
  • Stephan van Hulst
  • Tim Moores
  • Tim Holloway
  • Carey Brown
Bartenders:
  • Joe Ess
  • salvin francis
  • fred rosenberger

Advise on architecture-Data migration between two databases

 
Ranch Hand
Posts: 100
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi

I am having requirement to read data(some of the tables containing millions of records) from multiple tables(around 10 tables) in oracle then do some transformation and load this data onto cassandra. What would be the best way to do this.
Following is the approach that i can think of
1 Use apache camel for connecting to oracle, have separate camel routes to read data from each of the tables(if 10 tables then 10 routes will be defined). then drop these messages into kafka topics(data from each route will be dropped into separate kafka topic). Now there will be routes defined to read data from each of these kafka topics(10 consumers to consume from 10 kafka topics), do necessary transformation in batches using logic in processor(if required will execute processor in multi threaded mode) then connect to cassandra's keyspace and insert the data read.

Can you suggest me if this approach is a good way to perform data migration task between two databases when you have huge data. Will above approach have any scalability issues.
If there are any tweaks to be made to above approach or alternative frameworks best suited for this requirement please suggest.
Note: Source database can vary. In some cases it can be DB2
 
Saloon Keeper
Posts: 6060
154
Android Mac OS X Firefox Browser VI Editor Tomcat Server Safari
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
For any sizable extraction/transformation/load task I would recommend to use an ETL tool that is designed to do just that, rather than try to roll your own. Searching for oracle cassandra ETL tool will give you some starting pointers on that.
 
Doe, a deer, a female deer. Ray, a pockeful of sun. Me, a name, I call my tiny ad ...
Java file APIs (DOC, XLS, PDF, and many more)
https://products.aspose.com/total/java
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!