• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Opinions on best way to implement simple 1-way communication to threads

 
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I’m not sure if this belongs here or the Threads forum or the Sockets forum. Anyway…

Hi, all. I’m soliciting design opinions for this simple scenario:

I have an Ingest process that’s going to hang off of an ActiveMQ or RabbitMQ (it’s not important).

Ingest will consume every message it sees and parse each message for its type.
Ingest then forwards the message to a dedicated Type class/thread that does <something>.

Date rates are expected to be high. Ingest does not need to block when writing.
It just needs to shuffle the data as fast as possible to the appropriate Type “handler”.

The crux of my question has to do with how best to communicate the message to its class/thread.

My first attempt was to simply:

- Fire up Ingest
- Have Ingest instantiate the multiple Type objects
- Start each Type object in its own thread
- As each message comes into Ingest and is parsed for its type, call the appropriate Type object’s add() method
- add() adds the message to an internal container and then calls a process() method

The blatant problem with the above, of course, is with process():

If process() gets expensive then it will block add() and thus cause a block to backup into Ingest.

One solution to that (1.b) would be to have process() run in its own worker thread.
(Whether that still involves putting the entire object in a thread at startup as I said above, I don’t know.)
process() would then pop messages from the internal container one at a time and do <something>.

The other solution is to use some kind of queue for the communication between Ingest and the threads:

- Fire up Ingest
- Have Ingest instantiate the multiple Type objects
- Create and attach a queue of some kind (LinkedBlockingQueue?) to the Type object
- Start each Type object in its own thread
- As each message comes into Ingest and is parsed for its type, write the message to the appropriate Type object’s queue
- Let the Type object/thread do a blocking wait on the queue and process() each message as it comes in

This 2nd implementation seems like the cleaner solution. But I guess I wanted to see if I could avoid
any overhead the queues might introduce, and perform all the processing in the worker thread I mentioned
in implementation 1.b.

Thank you for any opinions or pointers.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
reply
    Bookmark Topic Watch Topic
  • New Topic