How to fix Kafka Broker may not be available on 127.0.0.1 error?

This is a common error when you start working with Kafka. Pretty much every Kafka developer has seen this error at least once.

Problem

You might encounter the below error message when your Kafka consumers or clients connect to Kafka broker for ingestion or consumption.

Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.

You might also see warnings like below

[2017-09-11 16:21:18,998] WARN Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) 

[2017-09-11 16:21:21,991] WARN Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

You could see this issue even when you properly refer to the Kafka brokers with their IP or hostname in your code or in your client scripts.

Solution

Pay attention to the below 2 properties in the <kafka-install-dir>/config/server.properties.

Under the Socket Server Settings – this setting controls where the host will listen to. The listener name (or host) is derived by Kafka by calling this Java method – java.net.InetAddress.getCanonicalHostName()

Based on how the host is configured this can return localhost or hostname or 127.0.0.1 

############################# Socket Server Settings #############################

# The address the socket server listens on. If not configured, the host name will be equal to the value of
# java.net.InetAddress.getCanonicalHostName(), with PLAINTEXT listener name, and port 9092.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092

Next, focus your attention to the advertised.listeners setting. This setting is very important. This setting controls how the broker advertises listener name, hostname and port to clients.

If not explicitly set, this will return the value of listeners property. If the value of the listeners property has the correct IP or the hostname of the broker, there won’t be any issues for the clients connecting to the broker.

But, if the value of the listeners property return 127.0.0.1 or localhost, then the client attempting connection to the broker will not find the broker on 127.0.0.1 or localhost because they refer to the host that the client or consumer is running on.

# Listener name, hostname and port the broker will advertise to clients.
# If not set, it uses the value for "listeners".
#advertised.listeners=PLAINTEXT://your.host.name:9092

Fix

Explicitly provide the hostname or ip address of the host of the broker in server.properties for both listeners and advertised.listeners.

listeners=PLAINTEXT://birw-node-1:9092

advertised.listeners=PLAINTEXT:///birw-node-1:9092

KRaft

This section doesn’t apply to you if you are not using KRaft.

Apache Kafka Raft (KRaft) is the consensus protocol that was introduced to remove Apache Kafka’s dependency on ZooKeeper for metadata management. This greatly simplifies Kafka’s architecture by consolidating responsibility for metadata into Kafka itself, rather than splitting it between two different systems: ZooKeeper and Kafka. KRaft mode makes use of a new quorum controller service in Kafka which replaces the previous controller and makes use of an event-based variant of the Raft consensus protocol.

Please note: KRaft is in early access and should be used in development only. It is not suitable for production.

If you are using KRaft, then you will see another server.properties under <kafka-install-dir>/config/kraft/server.properties

Explicitly set IP address or hostname in listeners and advertised.listeners in server.properties under kraft folder.