Hi all,
the functionality that i am trying to implement is:
1. Read file
2. validate each record(line).
3. store record to DB.
I want record processing should happen in parallel.
What I meant by this is, read a file by
thread A, handle the line(record) to sub thread to validate and store in DB), while sub thread busy doing validation and storing, thread A continues to read file.
Basically what I don't want to happend is,
read record, validate record and store record in sequential
pattern.
My initial sketch is something like below;
1. Create a Pool of threads
2. Create Jobqueue.
As the main thread start reading file, every record that fetches will put into, as and when records are available in the queue the second part of the process should get record from queue and validate it and store them , then pick next available record and continure untill it queue is empty.
Is this a right way of doing this, OR is there any better way of doing this?
If so, can some one here please suggest me.
any tools / opensource that has functionality of this kind is also welcome.
Bit of code snippet to get started with will be much appreciated.
Please help.
Thanks