Mike Curwen wrote:I'm looking for advice on the best approach to a problem like the following. I have an object that is responsible for retrieving rows from a database and doing quite a bit of processing for each entry. It also does further queries for each row, does a bit of templating and merging, potentially generating an email. Right now, the object just takes the selection criteria in the form of a key; work on all the rows that have 'this' value in the status field.
What if my rowcount is in the thousands? Right now it's just hundreds, and a single thread takes an acceptable amount of time. Eventually, I'm going to click a button, and it will take all night. So I want to multithread this...
My questions is.. where do the threads go?
Should I produce a 'Manager' object? Move the overall query from the existing class into this manager object, turn it into a SELECT COUNT(*) rather than SELECT *, have it divide by the 'maxThreads', and then instantiate 'maxThreads' instances of my (now slightly modified) processing object. Each modified object would then work on its own subset of rows. The only change in the existing processor object (I think) is the method signature would be two long ids, rather than one and the query would change from SELECT * where status=id to SELECT * where row_id between first_id AND last_id. Otherwise, all the code remains the same.
Forget the 'Manager' and simply implement threads within the existing class?
Is there a best practice here? Is there a better idea?
hmm... I just thought better of this. While currently, my rows are pretty much always in a contiguous block, this is not a guarantee. And the way MySql does LIMIT is not safe either:
If you use LIMIT # with ORDER BY, MySQL will end the sorting as soon as it has found the first # lines instead of sorting the whole table.
So I think I need to retrieve all the PK's and stuff them into a List. Then send sublists into the processor object. Yes?
[ December 31, 2003: Message edited by: Mike Curwen ]
Anton Golovin wrote:Hi, Mike,
Don't reinvent the wheel...