How to send large messages in Kafka?

By default the messages you can send and manage in Kafka should be less than 1 MB. To increase this limit there are few properties you need to change in both brokers and consumers.

Let’s say your messages can be up to 10 MB. So in this case your Kafka producers are producing messages up to 10 MB. So your Kafka Brokers and consumers should be able to store and receive messages up to 10 MB respectively.

Kafka Producer sends messages up to 10 MB ==> Kafka Broker allows, stores and manages messages up to 10 MB ==> Kafka Consumer receives messages up to 10 MB

Configuration on the broker side

Both the below properties need to be updated on the broker side to change the size of the message that can be handled by the brokers. Default for both the values is around 1 MB.

message.max.bytes

Maximum message size the broker accepts. When using the old consumer, this property must be lower than the consumer fetch.message.max.bytes, or the consumer cannot consume the message.

replica.fetch.max.bytes

Maximum message size a broker can replicate. Must be larger than message.max.bytes, or a broker can accept messages it cannot replicate, potentially resulting in data loss.

Configuration on the consumer side

fetch.message.max.bytes

This property is only applicable for older version of consumer (written in Scala). Default is 50 MB. If the amount of data returned by the broker for a consumer fetch exceeds the size specified in this property then the request would fail.

fetch.max.bytes

This property is only applicable for newer version of consumer (written in Java). Default is 50 MB. If the amount of data returned by the broker for a consumer fetch exceeds the size specified in this property then the request would fail.

Increase or change all the listed properties in this post appropriately if you are planning to increase the size of data to be managed by Kafka.