• Post Reply Bookmark Topic Watch Topic
  • New Topic

Achieving best performance while Iterating over the followers for a user  RSS feed

 
shivang sarawagi
Ranch Hand
Posts: 159
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am implementing a publish subscribe with Spring AMQP rabbit implementation.

On the consumer end the listener method receives the message from the queue. And then the logic is to send the message to the followers of the user.



Now assume a user has 100 thousand followers. What would be the best approach to handle this?

If the for loop runs a 100 thousand times, considering sending a notification to a follower takes 30 secs it will take a very long time to send a real time notification to all the followers

Or is there a way to iterate through a set of users, split the task into several threads using executor framework/Batch process or something?
 
Paul Clapham
Sheriff
Posts: 22831
43
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
shivang sarawagi wrote:Or is there a way to iterate through a set of users, split the task into several threads using executor framework/Batch process or something?


Sure, that sounds like a good idea. But really if you're going to have a system of that size then you aren't going to be running it on a single machine anyway, so there may be better ways. One important thing you should do is to reduce that 30 seconds to something more practical.
 
Campbell Ritchie
Marshal
Posts: 56546
172
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Depends what format you keep the followers in. If you can get them into a List, you can use a Stream for sequential access.
followersList.stream().map(f -> f.getAddress()).forEach(a -> sendMessage(a));

It is easy enough to change a Stream from sequential to parallel, by calling its parallel() method.
If you are using a single thread, you can use the List's forEach method directly.

This sort of code will only work in Java8. You can get Streams from all sorts of other things, e.g.a BufferedReader, or a file Path. Don't know whether you can get a Stream from an SQL connection.
 
Rob Spoor
Sheriff
Posts: 21135
87
Chrome Eclipse IDE Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Not out-of-the-box. I've searched which API methods return streams (for a presentation at work), and JDBC was not part of it (what would it return as stream elements?). FYI, Pattern is, with its splitAsStream method. For a full list just check the Use page of Stream: http://docs.oracle.com/javase/8/docs/api/java/util/stream/class-use/Stream.html
 
Campbell Ritchie
Marshal
Posts: 56546
172
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I couldn&apost find it in SQL classes by going through the API index for S. I forgot about the class‑use pages.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!