• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Understanding and learning Kafka

 
Ranch Hand
Posts: 2925
13
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I am learning Kafka. What I understand is that Kafka is used for ingesting huge amount of data at low latency. It is based on producer consumer pattern, where consumer subscribes to a topic and listens to it, and when producer produces message to that topic, it gets consumed by the consumer. What else should I be learning on Kafka? Thanks.
 
Saloon Keeper
Posts: 27811
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Well, the first thing to learn is that Kafka is neither Cloud nor Virtual. While it can be run on such platforms, it is something you can download and install locally as open-source software.

So I'm bouncing this the the "messaging" forum where it should be a better fit.
 
author
Posts: 23951
142
jQuery Eclipse IDE Firefox Browser VI Editor C++ Chrome Java Linux Windows
  • Likes 2
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Monica Shiralkar wrote:It is based on producer consumer pattern, where consumer subscribes to a topic and listens to it, and when producer produces message to that topic, it gets consumed by the consumer. What else should I be learning on Kafka?



This is very pedantic, but important to not get wrong. Apologies if you already know this.

In messaging, there is a difference between a producer and publisher, and also between a consumer and subscriber. Publishers and subscribers work with topics. And producers and consumers work with queues. Interestingly, I believe Kafka works with both -- so, you may confuse your colleagues, if you mix terms.

Messages on topics are seen by every subscriber -- meaning as a subscriber on a topic, you will see all messages from the publishers. Messages on queues are supposed to be delivers to only one consumer -- meaning as a consumer you will see a mix of messages from the producers (that is supposed to be unique to you). There is also rules on redelivery, if you don't acknowledge in time. Also, many brokers/managers can translate between topics and queues, so it can get more confusing.
 
Monica Shiralkar
Ranch Hand
Posts: 2925
13
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:Well, the first thing to learn is that Kafka is neither Cloud nor Virtual. While it can be run on such platforms, it is something you can download and install locally as open-source software.

So I'm bouncing this the the "messaging" forum where it should be a better fit.



Thanks. Yes, realized it now. It has nothing to do with Cloud. AWS cloud has a managed Kafka called "MSK".

For some reason, it is still showing up in Cloud/Virtualization forum instead of Messaging forum.
Staff note (Liutauras Vilda) :

It has been removed from Cloud/Virtualization forum. So now only in Messaging.

 
Monica Shiralkar
Ranch Hand
Posts: 2925
13
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks for correcting that.

Henry Wong wrote:
In messaging, there is a difference between a producer and publisher, and also between a consumer and subscriber. Publishers and subscribers work with topics. And producers and consumers work with queues.



When I simply create a Kafka topic, start a producer and consumer,  send a message from producer, I can see the message on consumer.

Here, it's producer - consumer (and no subscribe has been done).

It contradicts "Publishers and subscribers work with topics. And producers and consumers work with queues", since here I am using producer - consumer which is working with topic (not queue).

Or do you mean to say that "producer-consumer work on topics ?



 
Bartender
Posts: 2419
13
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Monica,
Do you find this comparsion between RabbitMQ and Kafka helpful:
https://www.simplilearn.com/kafka-vs-rabbitmq-article#:~:text=RabbitMQ%20sends%20messages%20to%20users,until%20the%20retention%20time%20expires. ?
 
Monica Shiralkar
Ranch Hand
Posts: 2925
13
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Himai Minh wrote:
Do you find this comparsion between RabbitMQ and Kafka helpful:
https://www.simplilearn.com/kafka-vs-rabbitmq-article#:~:text=RabbitMQ%20sends%20messages%20to%20users,until%20the%20retention%20time%20expires. ?



Thanks. This post says "Kafka is a pub/sub message bus", but when we create a Kafka topic, start a consumer for the topic, send messages from producer to this topic, and the messages get consumed by the consumer, isn't it producer-consumer and not publisher subscriber ?
 
Himai Minh
Bartender
Posts: 2419
13
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I think in Kafka, producer and publisher are used interchangeably and so do consumer and subscriber.
https://medium.com/@ujjawal.dixit/kafka-101-broker-producers-consumers-topics-and-partitions-b9bcbbdbf080
 
reply
    Bookmark Topic Watch Topic
  • New Topic