What is the difference between map and mapValues functions in Spark?
April 13, 2023How to rename an S3 bucket?
April 20, 2023Sometimes we have the need to see the message that we ingest into Kafka. This is typically needed for a quick sanity check or we recently made a change to the schema of the message and we want to make sure the change took effect properly.
In this post we will describe how to see messages that are ingested into a Kafka topic.
Kafka-console-consumer
Kafka out of the box comes with a handy script named kafka-console-consumer.sh. The script will read the messages from a topic and will print the messages on the screen.
The script, at the very least takes 3 inputs-
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --bootstrap-server - to connect to Kafka brokers --topic - name of the Kafka topic –-from-beginning - indicates that we would like to read messages from the very beginning
Read from specific offset and partition
If we would like to read messages from a specific offset and from a very specific partition, then we should specify offset and partition values in the command like below.
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test –partition 1 –offset 10
Limiting number of messages
Use the –max-messages flag to limit the number of maximum messages we would like to read.
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test –partition 1 –offset 10 –max-messages 150
