Win a copy of Murach's Python Programming this week in the Jython/Python forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

Using CyclicBarrier  RSS feed

 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi

I have several threads running simultaneously
I wish to pause the code in each at the same spot until a variable integer has reached a certain number

I see the code for CyclicBarrier but is it overkill ?

i.e. is there a simple approach to this pausing until.......

Bob M
 
Campbell Ritchie
Sheriff
Posts: 53759
127
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Don't know. Have added this discussion to our threads forum, and moving from beginning forum.
 
Stephan van Hulst
Saloon Keeper
Posts: 6980
110
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
If the variable is simply a count of the amount of threads that should wait before they all continue, CyclicBarrier is perfect! If the count-down is triggered by another thread somewhere else, use a CountDownLatch. If the integer doesn't representing a counting meachanism, you can use the more flexible Condition.
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Stephan

Each thread broadcasts a message to all the other threads

I wish ALL threads to wait until ALL threads have received all possible messages before proceeding

so you are saying that CyclicBarrier is the choice ?

Regards

Bob M
 
Tony Docherty
Bartender
Posts: 3201
78
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
A CyclicBarrier will block all threads calling it's await() method until the requisite numbers of threads have called await(). In the scenario you have just described this will probably cause your application to deadlock as all the messaging threads will block on the first other thread they send a message to. A CountDownLatch on the other hand will block the one thread that calls await() until the requisite number of calls to countdown() have occurred. In your scenario (ie the requisite number of messaging threads have called countdown())
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Tony

I am getting a little confused now

Lets say I have 5 threads running simultaneously
Every 6 hours they activate and start processing the code
Some way thru they send 1 message only to all the other threads
At this point they then WAIT

Each will begin with no messages received and slowly increase the counter until they have received 4 messages (from the other 4 threads)

Then the WAIT is over and they all continue processing their remaining code

BUT you feel that CyclicBarrier will deadlock ?

Bob M
 
Stephan van Hulst
Saloon Keeper
Posts: 6980
110
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yes, you can easily do this with a CyclicBarrier. Just initialize it with the number of threads, and then have each thread await the barrier after broadcasting their message. The barrier will automatically let all the threads pass as soon as all of them have reached the await() part.
 
Tony Docherty
Bartender
Posts: 3201
78
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
It depends on how you implement it.
I was thinking each thread would have it's own sync mechanism and so would notify all other thread of it's message and then wait. In this scenario if the sync mechanism was a CyclicBarrier then for each thread each message received would have to wait until all messages had been received which will potentially cause a deadlock whereas a CountDownLatch would work perfectly well. However, if implement a solution where you have a single global CyclicBarrier then each thread could wait on the global barrier after sending it's notifications.
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Stephan & Tony

My setup would be that each separate thread would have its own identical CyclicBarrier code.

NOT one global CyclicBarrier.

so are we still OK with that ?

Bob M
 
Tony Docherty
Bartender
Posts: 3201
78
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
If each thread has it's own sync code I would use a CountDownLatch for the reasons I have previously given.
 
Stephan van Hulst
Saloon Keeper
Posts: 6980
110
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Well, you should instantiate them all with a task that has access to the same barrier. Here's an example:



[edit]

In case you're not facing the scenario Tony described, of course.
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
My scenario which I shall try to explain clearly...................

I have a master control strategy which performs certain calculations (hourly)
I also have several cloned trading strategies - each a different currency pair - also (hourly)

I wish to implement two delays:-

(1) the master strategy WAITS until it has received a broadcast message from EACH and EVERY trading strategy - before continuing
(2) each trading strategy, after broadcasting the message in (1) above, WAITS until it receives a return message from the master strategy - before continuing

I feel that (1) can be done with CountDownLatch
and (2) can be down with CyclicBarrier

One thing I am struggling with is exactly in which programs do I put the required code?

Bob M

p.s. can I simply use wait() & notify() / notifyAll() ?
 
Stephan van Hulst
Saloon Keeper
Posts: 6980
110
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
They should both use a CountDownLatch. CyclicBarriers block all threads interacting with them. CountDownLatches can be controlled from threads that aren't blocked by them.

The idea is that every trading thread waits on the master latch until the master thread broadcasts its message, and then releases the master latch. The master latch then goes on to wait for the trading latch, which the trading threads release when they're done.

I'm not quite sure though why the trading threads should have to wait on the master thread, if the master thread can just spawn tasks and then wait for them to finish. You can do all of this using one shared CountDownLatch and a ScheduledExecutorService.
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Stephan

Thank you for your assistance and suggestion

The trading threads need to wait because the master thread takes a prospective trade from each of the trading threads, and then conducts a reduction process whereby say 10 prospective trades can be reduced to just 2 trades which achieve the identical result

Kind regards from New Zealand

Bob M
 
Stephan van Hulst
Saloon Keeper
Posts: 6980
110
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yes, but why don't they just finish completely, and let the master thread spawn new trading tasks for the next hour? Here's how I would do it:

 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Stephan

Many, many thanks for the suggested code..............

I shall come back to you when I have studied it fully and understand what it is doing...

Regards

Bob M
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Stephan

1) my strategies are actioned every 6 hours on the dot at 0, 6, 12, & 18pm
each strategy then calculates a prospective trade in the form of a currency pair, direction and size
That info is then sent to the master program and then the strategy waits

2) the master program waits until it has received x messages, where x = no. of trading strategies
Then it does the reduction process - say 10 trades down to 2
It then sends the resulting 2 trades info back to all the trading strategies
Each will look to see if it is 'on the list'
If it is it will initiate a trade - if not - it will do nothing for another 6 hours

I am very unclear as to how a ScheduledExecutorService would be in sync with the strategies ?

Bob M
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
How about this simple approach for each of the 2 required WAITS........................
 
Stephan van Hulst
Saloon Keeper
Posts: 6980
110
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Nope, that's what we call a "busy wait". Busy waits are bad because they hog processor power without doing anything useful. Using methods like Thread.sleep() or Thread.yield() are an indication of poor concurrency.

That info is then sent to the master program and then the strategy waits

This is important. Why does it wait? Why do the calculation of the prospective trade and making the actual trade have to be done in the same task or even the same thread? They are different tasks, so make them different tasks. A strategy is not a task. A strategy consists of tasks.

I am very unclear as to how a ScheduledExecutorService would be in sync with the strategies?

Because the master spawns the tasks, and then they report to the master and finish running. There is no need to be in sync, because they are short-lived.
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
OK - my bad suggestion

The strategies WAIT because they need to know what the list of optimized trades are (i.e. the reduced list) and it is the MASTER strategy which is calculating that info and broadcasting it back to each and every TRADING strategy

An alternative approach would be to have the reduction process (i.e. 10 prospective trades reduced to 2 optimized trades) done by ALL Trading strategies instead of being done by the MASTER strategy
i.e. each Trading strategy carries out the same calculations

In this instance, ONLY 1 WAIT is required
After each TRADING strategy has broadcasted its prospective trade to all the other TRADING strategies it WAITS until it has received x-1 messages (one from each of the other trading strategies where x = no. of trading strategies)

If we take this approach - would that simplify the require code ?

Bob M

p.s. at the moment, each TRADING strategy is independent of the others - it predicts a prospective trade and opens that trade itself, so I can have say 10 trading strategies all running simultaneously
 
Stephan van Hulst
Saloon Keeper
Posts: 6980
110
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

 
Stephan van Hulst
Saloon Keeper
Posts: 6980
110
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
1) All getProspect() calls are run concurrently.
2) Master finds an optimal list when all getProspect() calls are done.
3) All trade() calls are run concurrently. The all have access to the same optimal list.
4) When all trades are done, Master shuts down the executor, and everything is finished.
5) All that's left to do is schedule Master to run whenever you want.
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Stephan

I am still wrestling with my 'WAIT' requirements

Some MASTER code:-


Question 1: do I insert your recent MASTER code where indicated?

Some CLIENT code:-


Question 2: do I insert your recent CLIENT code where indicated?

Kind regards from New Zealand

Bob M
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Stephan

I am sorry but I can not use the code you suggested as I do not understand 90% of it

I really need a line by line explanation of the code.......

The optimized trades that result from the reduction process that MASTER performs once it has received all the prospective trades are NOT necessarily a subset of those trades
They will involve the same currency pairs that are handled by the various AGENTS but the size may be completely different to the size submitted as a prospective trade
In addition, the direction may or may not be in the same direction (i.e. Up or Down)

Bob M

 
Stephan van Hulst
Saloon Keeper
Posts: 6980
110
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yeah, but that doesn't matter. The findOptimal() in my code was just an example, you could apply any sort of mapping or reduction to the list of prospective trades, and pass the result back to the strategies.

I think you are too concerned with aspects of concurrency. First solve this problem with regular code, and afterwards identify areas that can be optimized with multi-threading:



This is roughly my understanding of your problem. Is it correct? If so, implement this in code, and stop worrying about threads, executors, and waiting. Every agent will have to wait for another to be done, but who really cares. We can optimize later. As long as the result is correct. Do this and I'll show you how to change your code to use multi-threading.
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Stephan

Thank you very much for your continued assistance - you must be very patient

Yes - I think your simplified outline is correct

Now, to use regular code ..............

I currently have each agent coded to
(a) perform every six hours
(b) to produce a prospective trade for the next 6 hours
So far so good..............

Add the prospective code to a master list
I can do that simply within each agent but how to share this list?

I have a Master program which also kicks-in every six hours in sync with the agents and includes the code for the transformation
and it works perfectly and produces the optimal list I am seeking

Not clear how to advise all agents of this result (i.e. the optimized list)?

Once each agent has the optimized list, the code from there on is OK

Hope this makes sense

Bob M

p.s. I could put the transformation code into each agent and forget about a Master program all together
then my only basic problem is how to advise each and every agent what all the prospective trades are

pp.s. currently the Agent programs have about 2,000 lines of code
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi again

Just to add to my confusion.......................

You say, forget about multi-threading for the time being BUT as I understand it I need to introduce socket protocol to pass the messages back and forth between the MASTER program and the various AGENT programs.

And it is this code which needs tightening up somewhat

Bob M
 
Stephan van Hulst
Saloon Keeper
Posts: 6980
110
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Haha, I understand now. You never mentioned they were different programs. Communication between programs is different from communication between tasks. I understand the requirement to wait now.

Still... Why are all the different agents different instances of a program? Do they have to run on separate machines?
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Well, I have different programs because.................

Each agent's program calculates a series of technical analysis indicators (moving averages, rates of change, etc. etc.)
and each agent calculates a different set of these.

Each agent's program carries out data mining (using WEKA) and each agent uses different WEKA attributes
The data mining involves a derby database - one for each agent - containing past trade details

In the future, with more agents being added, it would probably be useful to run so many agents on different machines situated in different places
Maybe, all running on a server in Switzerland, in Geneva where Dukascopy is

Bob M
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi again

The console shows the following for the MASTER program:-

2015-10-20 00:00:51 Message Received: EUR/JPY -2
2015-10-20 00:00:48 Message Received: USD/CHF 2
2015-10-20 00:00:47 Message Received: CHF/JPY 2
2015-10-20 00:00:39 Message Received: GBP/USD -2
2015-10-20 00:00:29 Message Received: USD/JPY 2

i.e. it has received all five prospective trades from the five agents that I have concurrently running

Now I need the MASTER to know that all five have been received and to proceed............

A portion of the socket MASTER code is as follows:-



the concept being that when the no. messages = no. agents, then MASTER continues processing.........

Bob M

[Edit] CR: have broken the very long line in your method heading. You can see how it should be done.
 
Stephan van Hulst
Saloon Keeper
Posts: 6980
110
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I think the code is too dense. It's hard to know what line is part of what responsibility, because so much is going on at the same time. Divide your responsibilities. The master doesn't need to know about streams, sockets, etc. etc.

All the master needs to know about are agents. Agent can be an interface that is used in both your master application as the different agent applications. The master just calls methods on the agents. If they are remote, the agents in the master application can delegate the calls using sockets, or even be implemented using RMI or some sort of web framework.





Now, on the other side, you write a class that receives messages coming from the SocketAgent of the master, delegates them to a LocalAgent, and then returns the result back to the master.

Please don't do smart stuff like: This code is unclear, unmaintainable and it WILL get you into trouble. Instead, use strongly typed Message instances that you serialize and deserialize. You can then do something like:Note that RMI can take all of this pain out of your hands.


Again, don't worry about all the SocketAgents running at the same time, let the Master interact with them one by one, and wait for their results one by one. Later you can focus on having the agents run concurrently.
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Stephan

Thank you again for all your advice
I love the way you keep introducing me to java apps that I know nothing about e.g. RMI

I have started to read about RMI............

The first thing I need to do is to agree upon an interface - i.e. a description of the methods I will allow remote clients to invoke.

My answer:-

A method that accepts as a parameter a string (namely a prospective trade) from each and every client, then calculates a list of optimized trades and returns this list to each and every client

Am I on the correct path ?

Bob M

p.s. by the way I am 68 years young so please forgive my slow uptake
 
Stephan van Hulst
Saloon Keeper
Posts: 6980
110
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Strings are very poor representations of data. They're not semantic, they're essentially just "sacks of information". They don't express what the data represents. Why not pass a strongly typed ProspectiveTrade?

Don't worry about the uptake, I'm enjoying this discussion, and I'm happy when people are open to new ideas
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi

My programs are full of strings

Please explain what a strongly typed ProspectiveTrade looks like
 
Stephan van Hulst
Saloon Keeper
Posts: 6980
110
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
It reifies whatever information you store in your string. Can you give an example of what such a string would look like?
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
A Prospective Trade looks like............

USDJPY 2

or GPBCHF -1

where xxxxxx = a currency pair
and n = 2, 1, 0, -1, or -2
 
Campbell Ritchie
Sheriff
Posts: 53759
127
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
No, that is its text representation, surely. It would be an object with CurrencySymbol fieldsYou will need a lot of verification in the constructors. You can probably easily make that class immutable.
 
Stephan van Hulst
Saloon Keeper
Posts: 6980
110
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
And this is exactly why String is a poor type. Without your explanation, "USDJPY 2" would have meant absolutely nothing to me. I'm assuming (again, I don't know, because there's no semantic value in a String) that the number is some sort of indication of how favorable the trade is? In that case, ProspectiveTrade would look like this:




US Dollar
Japanese Yen
HIGHLY_FAVORABLE
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
the integer [-2, -1, 0, 1, 2] represents the size or amount of trade and the sign represents the direction

2 = standard size
1 = 1/2 the standard size
0 = don't trade

plus sign = BUY
neg sign = SELL

At the end of the day, an optimized trade is no different from a prospective trade EXCEPT that abs(n) may be > 2 and n<>0

so perhaps we just define TRADE which has subset ProspectiveTRADE and OptimizedTRADE

The number of prospective trades = no. of clients and may include a 'no trade' i.e.n=0
The number of optimized trades <= no. of clients and will NOT include any 'no trades'

I am still puzzled over where and how the required delays are coded?
 
Bob Matthews
Ranch Hand
Posts: 136
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thinking of sockets and queues now.....................

I need to setup two queues:-

Queue1 - can have the following - subscribe, unsubscribe or a Prospective Trade [client to server]

Queue2 - will only have a list of optimized trades i.e. OptTrade1, OptTrade2 etc. [server to clients]

Sockets - each client will be connected to the server

I need to consider IP addresses, host names, port numbers, buffers, streams & blocking( avoiding deadlocks)
 
Consider Paul's rocket mass heater.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!