Win a copy of Rust Web Development this week in the Other Languages forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Tim Cooke
  • Campbell Ritchie
  • Ron McLeod
  • Liutauras Vilda
  • Jeanne Boyarsky
Sheriffs:
  • Junilu Lacar
  • Rob Spoor
  • Paul Clapham
Saloon Keepers:
  • Tim Holloway
  • Tim Moores
  • Jesse Silverman
  • Stephan van Hulst
  • Carey Brown
Bartenders:
  • Al Hobbs
  • Piet Souris
  • Frits Walraven

Storm : Max Spout Pending VS (KafkaConfig.bufferSizeBytes & KafkaConfig.fetchSizeBytes)

 
Greenhorn
Posts: 10
Eclipse IDE Python Spring
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi All,

When I Increase the KafkaConfig.bufferSizeBytes & KafkaConfig.fetchSizeBytes values to (100 * 1024 * 1024), I get the desired throughput, but the spout after sometime or whenever I restart the topology goes OUT OF MEMORY, but when I reduce the value of Bytes to (5 * 1024 * 1024) then it impacts the throughput largely.

I also wanted to know the relation between Max Spout Pending VS (KafkaConfig.bufferSizeBytes & KafkaConfig.fetchSizeBytes), does batch size is decided with Max Spout Pending Config or with KafkaConfig.bufferSizeBytes.

Regards,
Nilesh Chhapru.
 
Author
Posts: 14
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Max spout pending is the maximum number of tuples that can be in your topology unacknowledged from that spout at a given time.

Think of it this way.

Max Spout Pending is the cap equal to calls to next tuple on your spout minus tuples fully acked

So if your max spout pending is 10. You can have next tuple call 10 times without acking any of those tuples.

Your spout could be holding more messages from your broker waiting for next tuple to be called.

What you pull over from your broker at any time is a separate performance concern than the number of tuples that are "live" in your topology at any given period of time.

Max Spout Pending is a (simple) means of providing backpressure.
 
Nilesh Chhapru
Greenhorn
Posts: 10
Eclipse IDE Python Spring
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Sean,

Thanks for replying.

Yes, what you said is correct and is as per the documentation of storm, but when you have less buffer size and fetch size.. and the message you are trying to process is huge in size.. the you max spout pending wont work as per you have configured, since the buffer size is small and can pass only one or max 2 messages at one time.

I just want to know if we can draw a mathematical relation between both the configurations.

Regards,
Nilesh Chhapru.
 
Sean Allen
Author
Posts: 14
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You can either allot more memory per worker or work on improving the throughput of your bolts.

 
reply
    Bookmark Topic Watch Topic
  • New Topic