Win a copy of Murach's MySQL this week in the JDBC and Relational Databases forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Achieving best performance while Iterating over the followers for a user

 
Ranch Hand
Posts: 159
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I am implementing a publish subscribe with Spring AMQP rabbit implementation.

On the consumer end the listener method receives the message from the queue. And then the logic is to send the message to the followers of the user.



Now assume a user has 100 thousand followers. What would be the best approach to handle this?

If the for loop runs a 100 thousand times, considering sending a notification to a follower takes 30 secs it will take a very long time to send a real time notification to all the followers

Or is there a way to iterate through a set of users, split the task into several threads using executor framework/Batch process or something?
 
Marshal
Posts: 28127
94
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

shivang sarawagi wrote:Or is there a way to iterate through a set of users, split the task into several threads using executor framework/Batch process or something?



Sure, that sounds like a good idea. But really if you're going to have a system of that size then you aren't going to be running it on a single machine anyway, so there may be better ways. One important thing you should do is to reduce that 30 seconds to something more practical.
 
Marshal
Posts: 79022
375
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Depends what format you keep the followers in. If you can get them into a List, you can use a Stream for sequential access.
followersList.stream().map(f -> f.getAddress()).forEach(a -> sendMessage(a));

It is easy enough to change a Stream from sequential to parallel, by calling its parallel() method.
If you are using a single thread, you can use the List's forEach method directly.

This sort of code will only work in Java8. You can get Streams from all sorts of other things, e.g.a BufferedReader, or a file Path. Don't know whether you can get a Stream from an SQL connection.
 
Sheriff
Posts: 22778
130
Eclipse IDE Spring VI Editor Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Not out-of-the-box. I've searched which API methods return streams (for a presentation at work), and JDBC was not part of it (what would it return as stream elements?). FYI, Pattern is, with its splitAsStream method. For a full list just check the Use page of Stream: http://docs.oracle.com/javase/8/docs/api/java/util/stream/class-use/Stream.html
 
Campbell Ritchie
Marshal
Posts: 79022
375
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I couldn&apost find it in SQL classes by going through the API index for S. I forgot about the class‑use pages.
 
reply
    Bookmark Topic Watch Topic
  • New Topic