I have an extremely high performance high scale high available problem. It's sort of combining producer/consumer, broadcast/subscribe in a very high volume basis. Think 10k transactions per second with responses needed in 50msecs in a 24x7x365 up time scenario.
I have a pool of producers spread across multiple servers that get work. Ideally each producer would get a
unit of work and broadcast the job out to all consumers of different types.
1. A Producer instance broadcasts job to consumer type A, consumer type B, consumer type C spread across multiple servers.
2. Consumer type A, B, and C work on the problem and respond with their individual results.
3. There will be many instances of consumer type A (or B or C), but only 1 instance should work on any one job.
4. Producer instance waits for results from A, B, and C and then picks the most optimal result from the work done by A, B, and C. If A, B, or C are slow to respond, they lose (expiry maybe?).
5. Producers and consumers will all be in multiple servers (VIPs).
What are the best technologies to consider for this?