Asynchronous Event Handling Using Microservices and Kafka

[Brian Mericle]

Hello Everyone,

My Manning liveProject Asynchronous Event Handling Using Microservices and Kafka is on promotion starting tomorrow (8/10/21). I'll be around the whole week to answer questions or participate in discussions about it.

[Campbell Ritchie]

Hope to see you there Welcome

[Kelly X.Y.]

What is the good way to trace a message in its message path in an asynchronous event-driven Microservice architecture?

[tangara goh]

Hi Brian,

I would like to know if it is possible to use Kubernetes to deploy Kafka applications in a Microservices setting.

Tks.

[Nitish Singla]

Kelly X.Y. wrote:What is the good way to trace a message in its message path in an asynchronous event-driven Microservice architecture?

I can think of a module from spring, spring - sleuth it is a tool for tracing logs in a distributed environment

[Nitish Singla]

Hey Brian

I am excited to read your book. So far have tried few things with Kafka along with Spring.
Few problems faced are Kafka lag, and I wonder how can we fastforward the offset if needed be?
How to effectively make use of concurrency factor to process the messages faster.

Also had another question around processing a message only once, I have a scenario where have written db triggers to which our application is listening. And since we have multiple instances of application running. Each instance processes the trigger and submits it to Kafka topic.  How can we make sure they message is written only once?

Regards
Nitish

[Brian Mericle]

Kelly X.Y. wrote:What is the good way to trace a message in its message path in an asynchronous event-driven Microservice architecture?

Hi Kelly! One way I have used to trace incoming messages throughout its lifecycle within a system is to add a correlation id to the message. This could be auto-generated upon receipt at the first entry point, or the client publishing the message could provide one. Many languages support creating a UUID, and because they are unique, make great identifiers. I would then, at every boundary in the system that interacts with the message, write a log message containing the correlation id and any other relevant information you would want to know about the message at that given time. One requirement of this approach would be to ensure that the correlation id would need to be passed across each boundary.

As Nitish mentioned in his post, you could use a library like Spring Cloud Sleuth that provides distributed tracing around ingress and egress for some top level components. It can help collect logs related to a request and measure the execution time (which is helpful to troubleshoot bottlenecks in the system).

If you are already using a monitoring solution like DataDog or AppDynamics, they have Application Performance Monitoring (APM) capabilities that you could use to instrument your code for traceability.

[Brian Mericle]

tangara goh wrote:Hi Brian,

I would like to know if it is possible to use Kubernetes to deploy Kafka applications in a Microservices setting.

Tks.

Hi Tangara! Absolutely you can run Kafka in Kubernetes. Because of the persistent storage requirements for Kafka, there isn't an easy way to install and maintain it unless you are using a package manager like Helm. If you are using that, then you could use the Bitnami Helm Chart. If you are using Confluent.io then they provide a Kubernetes Operator.

You could also run Kafka easily and locally for development purposes, which I cover in the first Milestone in my liveProject.

[Brian Mericle]

Nitish Singla wrote:Hey Brian

I am excited to read your book. So far have tried few things with Kafka along with Spring.
Few problems faced are Kafka lag, and I wonder how can we fastforward the offset if needed be?
How to effectively make use of concurrency factor to process the messages faster.

Also had another question around processing a message only once, I have a scenario where have written db triggers to which our application is listening. And since we have multiple instances of application running. Each instance processes the trigger and submits it to Kafka topic.  How can we make sure they message is written only once?

Regards
Nitish

Hi Nitish! First, thank you for your suggestion with regard to Kelly's question. These are some great questions! I would highly recommend reviewing the Kafka Consumer Javadocs. There is some great information in there about not relying on automatic offset commits and allowing the consumer to either re-read older messages, or skipping ahead to the most recent. The KafkaConsumer is not thread-safe, so make sure to read the section on Multi-threaded Processing in that link to the Javadocs.

As far as only-once processing is concerned, there is a Transactional API available that can help ensure exactly-once delivery between a producer and consumer. Here is a great example of using this approach.

[Brian Mericle]

Hi Everyone!

I thought I'd explain a little bit about my liveProject. There have been many references to this project in these threads about being a "Book". A Manning liveProject is very different in that it provides the opportunity to learn new skills by completing real-world challenges in your local development environment. It is essentially a hands-on course that is way more involved than a typical "Hello World" tutorial. The scenarios provide opportunities to solve practical problems while writing working code.

In this liveProject, you’ll use the Kafka distributed streaming data platform to help convert a legacy order fulfillment system to a sleek new asynchronous event-driven architecture. This architecture needs to deliver services that can scale and evolve independently, to speed up your clients eCommerce app. You’ll use the Go language to build a microservice, an event publisher, and multiple event consumers; and work with Kafka’s command-line tools to create, alter, and test topics.

I hope you'll check out my Asynchronous Event Handling Using Microservices and Kafka liveProject! I'm also the mentor on the liveProject, so when you have questions, I'll be there to respond.