• Post Reply Bookmark Topic Watch Topic
  • New Topic

Opinions on best way to implement simple 1-way communication to threads  RSS feed

 
Ron Callahan
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I’m not sure if this belongs here or the Threads forum or the Sockets forum. Anyway…

Hi, all. I’m soliciting design opinions for this simple scenario:

I have an Ingest process that’s going to hang off of an ActiveMQ or RabbitMQ (it’s not important).

Ingest will consume every message it sees and parse each message for its type.
Ingest then forwards the message to a dedicated Type class/thread that does <something>.

Date rates are expected to be high. Ingest does not need to block when writing.
It just needs to shuffle the data as fast as possible to the appropriate Type “handler”.

The crux of my question has to do with how best to communicate the message to its class/thread.

My first attempt was to simply:

- Fire up Ingest
- Have Ingest instantiate the multiple Type objects
- Start each Type object in its own thread
- As each message comes into Ingest and is parsed for its type, call the appropriate Type object’s add() method
- add() adds the message to an internal container and then calls a process() method

The blatant problem with the above, of course, is with process():

If process() gets expensive then it will block add() and thus cause a block to backup into Ingest.

One solution to that (1.b) would be to have process() run in its own worker thread.
(Whether that still involves putting the entire object in a thread at startup as I said above, I don’t know.)
process() would then pop messages from the internal container one at a time and do <something>.

The other solution is to use some kind of queue for the communication between Ingest and the threads:

- Fire up Ingest
- Have Ingest instantiate the multiple Type objects
- Create and attach a queue of some kind (LinkedBlockingQueue?) to the Type object
- Start each Type object in its own thread
- As each message comes into Ingest and is parsed for its type, write the message to the appropriate Type object’s queue
- Let the Type object/thread do a blocking wait on the queue and process() each message as it comes in

This 2nd implementation seems like the cleaner solution. But I guess I wanted to see if I could avoid
any overhead the queues might introduce, and perform all the processing in the worker thread I mentioned
in implementation 1.b.

Thank you for any opinions or pointers.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!