I am doing sync between sql DB to vertica.
I am moving 5 giga data in the network from sql and insert it in our document structure in mongo.
i am doing it with 10 connections to the mongo and 10 threads.
everything is going well but after i copy 1 giga, the rate of the copy become slow more and more until i dont see more new docs in the collection.
again 10 threads every one has a connection to Vertica DB(sql) and to the mongo.each one of the threads connect to Vertica copy some data turn it into entities and save it in Mongo.
they run and after ~1 giga the process become very slow until i dont see any more new docs in the collection(and there are more for sure
).
do you familiar with this?
can you advise?