How to fix the LEADER_NOT_AVAILABLE error in Kafka?

If you are using Kafka on the public cloud like AWS or Azure or on Docker you are more likely to experience the below error.

WARN Error while fetching metadata with correlation id 39 : {sales-topic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

Cause of LEADER_NOT_AVAILABLE error

As a producer or consumer in a Kafka setup, the broker hostname you pass will help you get the metadata of the other brokers in the cluster which are acting as leaders for the partitions you are trying to write to or read from.

Let’s say your brokers are set up in AWS EC2. AWS has a multi-layer networking setup so for every EC2 host you will have a public DNS (ec2-x-y-z-zz.us-west-2.compute.amazonaws.com) and a private DNS (ip-a-b-c-d.us-west-2.compute.internal). If you are a producer or consumer outside the Kakfa cluster network you will be able to only reach the brokers with their public DNS and not private DNS. Private DNS can only be used when your producer or consumer is on the same network as your Kafka brokers. 

Let’s assume your client, that is your producer or consumer is outside the network of your Kakfa cluster and in that case you will be able to reach the brokers only with their public DNS. The broker will return the private DNS of the brokers hosting the leader partitions and not the public DNS. Unfortunately since your client is not on the same network as your Kafka cluster, your client will not be able to  resolve the private DNS which will lead to LEADER_NOT_AVAILABLE error.

Solution to LEADER_NOT_AVAILABLE error

You might have seen that a fix is to update your host name mappings in /etc/hosts file and it is more of a hack than a real fix.

When you start Kafka brokers, the brokers can listen to multiple listeners. Listeners are nothing but hostname or IP, port and protocol combination. 

The below properties go in server.properties  file on each Kafka broker. Note that advertised.listeners  is the important property that will help you solve the issue.

• listeners  – comma separated hostnames with ports on which Kafka brokers listen to 
• advertised.listeners  – comma separated hostnames with ports which will be passed back to the clients. Make sure to include only the hostnames which will be resolved at the client (producer or consumer) side for eg. public DNS.
• listener.security.protocol.map  – includes the supported protocols for each listener
• inter.broker.listener.name  – listeners to be used for internal traffic between brokers. These hostnames included here don’t need to be resolved at the client side but need to be resolved at all the brokers in the cluster.

listeners: LISTENER_PUBLIC://kafka0:29092,LISTENER_INTERNAL://localhost:9092
advertised.listeners: LISTENER_PUBLIC://kafka0:29092,LISTENER_INTERNAL://localhost:9092
listener.security.protocol.map: LISTENER_PUBLIC:PLAINTEXT,LISTENER_INTERNAL:PLAINTEXT
inter.broker.listener.name: LISTENER_PUBLIC

Docker setup

Use the below properties if you are running your Kafka in Docker

KAFKA_LISTENERS: LISTENER_PUBLIC://kafka0:29092,LISTENER_INTERNAL://localhost:9092
KAFKA_ADVERTISED_LISTENERS: LISTENER_PUBLIC://kafka0:29092,LISTENER_INTERNAL://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_PUBLIC:PLAINTEXT,LISTENER_INTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_PUBLIC