• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Tim Cooke
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Liutauras Vilda
Sheriffs:
  • Rob Spoor
  • Junilu Lacar
  • paul wheaton
Saloon Keepers:
  • Stephan van Hulst
  • Tim Moores
  • Tim Holloway
  • Carey Brown
  • Scott Selikoff
Bartenders:
  • Piet Souris
  • Jj Roberts
  • fred rosenberger

Opinions on best way to implement simple 1-way communication to threads

 
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I’m not sure if this belongs here or the Threads forum or the Sockets forum. Anyway…

Hi, all. I’m soliciting design opinions for this simple scenario:

I have an Ingest process that’s going to hang off of an ActiveMQ or RabbitMQ (it’s not important).

Ingest will consume every message it sees and parse each message for its type.
Ingest then forwards the message to a dedicated Type class/thread that does <something>.

Date rates are expected to be high. Ingest does not need to block when writing.
It just needs to shuffle the data as fast as possible to the appropriate Type “handler”.

The crux of my question has to do with how best to communicate the message to its class/thread.

My first attempt was to simply:

- Fire up Ingest
- Have Ingest instantiate the multiple Type objects
- Start each Type object in its own thread
- As each message comes into Ingest and is parsed for its type, call the appropriate Type object’s add() method
- add() adds the message to an internal container and then calls a process() method

The blatant problem with the above, of course, is with process():

If process() gets expensive then it will block add() and thus cause a block to backup into Ingest.

One solution to that (1.b) would be to have process() run in its own worker thread.
(Whether that still involves putting the entire object in a thread at startup as I said above, I don’t know.)
process() would then pop messages from the internal container one at a time and do <something>.

The other solution is to use some kind of queue for the communication between Ingest and the threads:

- Fire up Ingest
- Have Ingest instantiate the multiple Type objects
- Create and attach a queue of some kind (LinkedBlockingQueue?) to the Type object
- Start each Type object in its own thread
- As each message comes into Ingest and is parsed for its type, write the message to the appropriate Type object’s queue
- Let the Type object/thread do a blocking wait on the queue and process() each message as it comes in

This 2nd implementation seems like the cleaner solution. But I guess I wanted to see if I could avoid
any overhead the queues might introduce, and perform all the processing in the worker thread I mentioned
in implementation 1.b.

Thank you for any opinions or pointers.
 
pie. tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
reply
    Bookmark Topic Watch Topic
  • New Topic