Kafka

Let’s say you have a consumer group which has 3 consumers at the moment consuming messages from a topic. Assume that you had to shut down […]

In modern software systems, data is often generated and consumed in real-time. To handle these data streams, various processing techniques have been developed, including stream processing […]

Kafka uses Zookeeper to manage it’s internal state. So it is not possible to run Kafka without Zookeeper. Even if you don’t have access to Zookeeper […]

This is a common error when you start working with Kafka. Pretty much every Kafka developer has seen this error at least once. Problem You might […]

Sometimes we have the need to see the message that we ingest into Kafka. This is typically needed for a quick sanity check or we recently […]

Changing the number of replicas on an existing topic is a 3 step process. Get the current information about the topic Configure new replica assignment in […]

In Kafka, consumers connected to a topic are grouped into consumer groups. So a consumer group will have a list of consumers. So to get the […]

Let’s redefine the question a little bit. Can multiple Kafka consumers from a consumer group read the same message from a partition? The short answer to […]

This is a very common use case. Consumer offset is used to track the messages that are consumed by consumers in a consumer group. A topic […]

There is a lot of confusion when it comes to consumer offset, what is the purpose for it and how to change the offset. The goal […]

By default the messages you can send and manage in Kafka should be less than 1 MB. To increase this limit there are few properties you […]

If you are using Kafka on the public cloud like AWS or Azure or on Docker you are more likely to experience the below error. WARN […]

Ever wondered the differences between Rabbit MQ and Kafka and the need for Kafka when there is already RabbitMQ? Read through the post and you will […]

Kafka  Distributed, durable and reliable message broker which can handle high volume of real time messages coming from realtime producers. Storage for real time streaming data […]

There are 2 ways to get the list of available brokers in a Kafka cluster. Both with the help of scripts from zookeeper. Zookeeper manages the […]

In this post we will see how to delete a Kafka topic and get the details of the topic before deleting it. Fetch current details of […]

The easiest way to purge or delete messages in a Kafka topic is by setting the retention.ms to a low value. retention.ms configuration controls how long […]