This week's book giveaway is in the Other Languages forum.
We're giving away four copies of Rust Web Development and have Bastian Gruber on-line!
See this thread for details.
Win a copy of Rust Web Development this week in the Other Languages forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Tim Cooke
  • Campbell Ritchie
  • Ron McLeod
  • Liutauras Vilda
  • Jeanne Boyarsky
Sheriffs:
  • Junilu Lacar
  • Rob Spoor
  • Paul Clapham
Saloon Keepers:
  • Tim Holloway
  • Tim Moores
  • Jesse Silverman
  • Stephan van Hulst
  • Carey Brown
Bartenders:
  • Al Hobbs
  • Piet Souris
  • Frits Walraven

Achieving best performance while Iterating over the followers for a user

 
Ranch Hand
Posts: 159
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I am implementing a publish subscribe with Spring AMQP rabbit implementation.

On the consumer end the listener method receives the message from the queue. And then the logic is to send the message to the followers of the user.



Now assume a user has 100 thousand followers. What would be the best approach to handle this?

If the for loop runs a 100 thousand times, considering sending a notification to a follower takes 30 secs it will take a very long time to send a real time notification to all the followers

Or is there a way to iterate through a set of users, split the task into several threads using executor framework/Batch process or something?
 
Sheriff
Posts: 26948
83
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

shivang sarawagi wrote:Or is there a way to iterate through a set of users, split the task into several threads using executor framework/Batch process or something?



Sure, that sounds like a good idea. But really if you're going to have a system of that size then you aren't going to be running it on a single machine anyway, so there may be better ways. One important thing you should do is to reduce that 30 seconds to something more practical.
 
Marshal
Posts: 74627
335
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Depends what format you keep the followers in. If you can get them into a List, you can use a Stream for sequential access.
followersList.stream().map(f -> f.getAddress()).forEach(a -> sendMessage(a));

It is easy enough to change a Stream from sequential to parallel, by calling its parallel() method.
If you are using a single thread, you can use the List's forEach method directly.

This sort of code will only work in Java8. You can get Streams from all sorts of other things, e.g.a BufferedReader, or a file Path. Don't know whether you can get a Stream from an SQL connection.
 
Sheriff
Posts: 22563
122
Eclipse IDE Spring VI Editor Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Not out-of-the-box. I've searched which API methods return streams (for a presentation at work), and JDBC was not part of it (what would it return as stream elements?). FYI, Pattern is, with its splitAsStream method. For a full list just check the Use page of Stream: http://docs.oracle.com/javase/8/docs/api/java/util/stream/class-use/Stream.html
 
Campbell Ritchie
Marshal
Posts: 74627
335
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I couldn&apost find it in SQL classes by going through the API index for S. I forgot about the class‑use pages.
 
You showed up just in time for the waffles! And this tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
reply
    Bookmark Topic Watch Topic
  • New Topic