Win a copy of Node.js Design Patterns: Design and implement production-grade Node.js applications using proven patterns and techniques this week in the Server-Side JavaScript and NodeJS forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Ron McLeod
  • Rob Spoor
  • Tim Cooke
  • Junilu Lacar
Sheriffs:
  • Henry Wong
  • Liutauras Vilda
  • Jeanne Boyarsky
Saloon Keepers:
  • Jesse Silverman
  • Tim Holloway
  • Stephan van Hulst
  • Tim Moores
  • Carey Brown
Bartenders:
  • Al Hobbs
  • Mikalai Zaikin
  • Piet Souris

Asynchronous Event Handling Using Microservices and Kafka

 
Author
Posts: 7
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello Everyone,

My Manning liveProject Asynchronous Event Handling Using Microservices and Kafka is on promotion starting tomorrow (8/10/21). I'll be around the whole week to answer questions or participate in discussions about it.

 
Marshal
Posts: 74084
332
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hope to see you there Welcome
 
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
What is the good way to trace a message in its message path in an asynchronous event-driven Microservice architecture?
 
Ranch Hand
Posts: 821
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Brian,

I would like to know if it is possible to use Kubernetes to deploy Kafka applications in a Microservices setting.

Tks.
 
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Kelly X.Y. wrote:What is the good way to trace a message in its message path in an asynchronous event-driven Microservice architecture?




I can think of a module from spring, spring - sleuth it is a tool for tracing logs in a distributed environment
 
Nitish Singla
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hey Brian

I am excited to read your book. So far have tried few things with Kafka along with Spring.
Few problems faced are Kafka lag, and I wonder how can we fastforward the offset if needed be?
How to effectively make use of concurrency factor to process the messages faster.

Also had another question around processing a message only once, I have a scenario where have written db triggers to which our application is listening. And since we have multiple instances of application running. Each instance processes the trigger and submits it to Kafka topic.  How can we make sure they message is written only once?

Regards
Nitish
 
Brian Mericle
Author
Posts: 7
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Kelly X.Y. wrote:What is the good way to trace a message in its message path in an asynchronous event-driven Microservice architecture?



Hi Kelly! One way I have used to trace incoming messages throughout its lifecycle within a system is to add a correlation id to the message. This could be auto-generated upon receipt at the first entry point, or the client publishing the message could provide one. Many languages support creating a UUID, and because they are unique, make great identifiers. I would then, at every boundary in the system that interacts with the message, write a log message containing the correlation id and any other relevant information you would want to know about the message at that given time. One requirement of this approach would be to ensure that the correlation id would need to be passed across each boundary.

As Nitish mentioned in his post, you could use a library like Spring Cloud Sleuth that provides distributed tracing around ingress and egress for some top level components. It can help collect logs related to a request and measure the execution time (which is helpful to troubleshoot bottlenecks in the system).

If you are already using a monitoring solution like DataDog or AppDynamics, they have Application Performance Monitoring (APM) capabilities that you could use to instrument your code for traceability.
 
Brian Mericle
Author
Posts: 7
6
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

tangara goh wrote:Hi Brian,

I would like to know if it is possible to use Kubernetes to deploy Kafka applications in a Microservices setting.

Tks.



Hi Tangara! Absolutely you can run Kafka in Kubernetes. Because of the persistent storage requirements for Kafka, there isn't an easy way to install and maintain it unless you are using a package manager like Helm. If you are using that, then you could use the Bitnami Helm Chart. If you are using Confluent.io then they provide a Kubernetes Operator.

You could also run Kafka easily and locally for development purposes, which I cover in the first Milestone in my liveProject.
 
Brian Mericle
Author
Posts: 7
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Nitish Singla wrote:Hey Brian

I am excited to read your book. So far have tried few things with Kafka along with Spring.
Few problems faced are Kafka lag, and I wonder how can we fastforward the offset if needed be?
How to effectively make use of concurrency factor to process the messages faster.

Also had another question around processing a message only once, I have a scenario where have written db triggers to which our application is listening. And since we have multiple instances of application running. Each instance processes the trigger and submits it to Kafka topic.  How can we make sure they message is written only once?

Regards
Nitish



Hi Nitish! First, thank you for your suggestion with regard to Kelly's question. These are some great questions! I would highly recommend reviewing the Kafka Consumer Javadocs. There is some great information in there about not relying on automatic offset commits and allowing the consumer to either re-read older messages, or skipping ahead to the most recent. The KafkaConsumer is not thread-safe, so make sure to read the section on Multi-threaded Processing in that link to the Javadocs.

As far as only-once processing is concerned, there is a Transactional API available that can help ensure exactly-once delivery between a producer and consumer. Here is a great example of using this approach.
 
Brian Mericle
Author
Posts: 7
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Everyone!

I thought I'd explain a little bit about my liveProject. There have been many references to this project in these threads about being a "Book". A Manning liveProject is very different in that it provides the opportunity to learn new skills by completing real-world challenges in your local development environment. It is essentially a hands-on course that is way more involved than a typical "Hello World" tutorial. The scenarios provide opportunities to solve practical problems while writing working code.

In this liveProject, you’ll use the Kafka distributed streaming data platform to help convert a legacy order fulfillment system to a sleek new asynchronous event-driven architecture. This architecture needs to deliver services that can scale and evolve independently, to speed up your clients eCommerce app. You’ll use the Go language to build a microservice, an event publisher, and multiple event consumers; and work with Kafka’s command-line tools to create, alter, and test topics.

I hope you'll check out my Asynchronous Event Handling Using Microservices and Kafka liveProject! I'm also the mentor on the liveProject, so when you have questions, I'll be there to respond.
 
reply
    Bookmark Topic Watch Topic
  • New Topic