{"id":2270,"date":"2023-05-08T06:00:00","date_gmt":"2023-05-08T11:00:00","guid":{"rendered":"https:\/\/www.bigdatainrealworld.com\/?p=2270"},"modified":"2023-04-18T19:26:33","modified_gmt":"2023-04-19T00:26:33","slug":"how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error","status":"publish","type":"post","link":"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/","title":{"rendered":"How to fix Kafka Broker may not be available on 127.0.0.1 error?"},"content":{"rendered":"\n<p>This is a common error when you start working with Kafka. Pretty much every Kafka developer has seen this error at least once.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Problem<\/h2>\n\n\n\n<p>You might encounter the below error message when your Kafka consumers or clients connect to Kafka broker for ingestion or consumption.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">Connection to node -1 (\/127.0.0.1:9092) could not be established. Broker may not be available.<\/pre>\n\n\n\n<p>You might also see warnings like below<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[2017-09-11 16:21:18,998] WARN Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) \n\n[2017-09-11 16:21:21,991] WARN Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)<\/pre>\n\n\n\n<p>You could see this issue even when you properly refer to the Kafka brokers with their IP or hostname in your code or in your client scripts.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Solution<\/h2>\n\n\n\n<p>Pay attention to the below 2 properties in the &lt;kafka-install-dir&gt;\/config\/server.properties.<\/p>\n\n\n\n<p>Under the Socket Server Settings &#8211; this setting controls where the host will listen to. The listener name (or host) is derived by Kafka by calling this Java method &#8211; java.net.InetAddress.getCanonicalHostName()<\/p>\n\n\n\n<p>Based on how the host is configured this can return localhost or hostname or 127.0.0.1\u00a0<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">############################# Socket Server Settings #############################\r\n\r\n# The address the socket server listens on. If not configured, the host name will be equal to the value of\r\n# java.net.InetAddress.getCanonicalHostName(), with PLAINTEXT listener name, and port 9092.\r\n#   FORMAT:\r\n#     listeners = listener_name:\/\/host_name:port\r\n#   EXAMPLE:\r\n#     listeners = PLAINTEXT:\/\/your.host.name:9092\r\n#listeners=PLAINTEXT:\/\/:9092\r<\/pre>\n\n\n\n<p>Next, focus your attention to the advertised.listeners setting. This setting is very important. This setting controls how the broker advertises listener name, hostname and port to clients.<\/p>\n\n\n\n<p>If not explicitly set, this will return the value of listeners property. If the value of the listeners property has the correct IP or the hostname of the broker, there won\u2019t be any issues for the clients connecting to the broker.<\/p>\n\n\n\n<p>But, if the value of the listeners property return 127.0.0.1 or localhost, then the client attempting connection to the broker will not find the broker on 127.0.0.1 or localhost because they refer to the host that the client or consumer is running on.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"># Listener name, hostname and port the broker will advertise to clients.\r\n# If not set, it uses the value for \"listeners\".\r\n#advertised.listeners=PLAINTEXT:\/\/your.host.name:9092\n<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Fix<\/h3>\n\n\n\n<p>Explicitly provide the hostname or ip address of the host of the broker in server.properties for both listeners and advertised.listeners.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">listeners=PLAINTEXT:\/\/birw-node-1:9092\r\n\r\nadvertised.listeners=PLAINTEXT:\/\/\/birw-node-1:9092\r\n<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">KRaft<\/h2>\n\n\n\n<p>This section doesn\u2019t apply to you if you are not using KRaft.<\/p>\n\n\n\n<p>Apache Kafka Raft (KRaft) is the consensus protocol that was introduced to remove Apache Kafka\u2019s dependency on ZooKeeper for metadata management. This greatly simplifies Kafka\u2019s architecture by consolidating responsibility for metadata into Kafka itself, rather than splitting it between two different systems: ZooKeeper and Kafka. KRaft mode makes use of a new quorum controller service in Kafka which replaces the previous controller and makes use of an event-based variant of the Raft consensus protocol.<\/p>\n\n\n\n<p>Please note: KRaft is in early access and should be used in development only. It is not suitable for production.<\/p>\n\n\n\n<p>If you are using KRaft, then you will see another server.properties under &lt;kafka-install-dir&gt;\/config\/kraft\/server.properties<\/p>\n\n\n\n<p>Explicitly set IP address or hostname in listeners and advertised.listeners in server.properties under kraft folder.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This is a common error when you start working with Kafka. Pretty much every Kafka developer has seen this error at least once. Problem You might<span class=\"excerpt-hellip\"> [\u2026]<\/span><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[15],"tags":[],"class_list":["post-2270","post","type-post","status-publish","format-standard","hentry","category-kafka"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How to fix Kafka Broker may not be available on 127.0.0.1 error? - Big Data In Real World<\/title>\n<meta name=\"description\" content=\"Pretty much every Kafka developer has seen this error at least once. This post discuss the error, reason and solution for the same.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to fix Kafka Broker may not be available on 127.0.0.1 error? - Big Data In Real World\" \/>\n<meta property=\"og:description\" content=\"Pretty much every Kafka developer has seen this error at least once. This post discuss the error, reason and solution for the same.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/\" \/>\n<meta property=\"og:site_name\" content=\"Big Data In Real World\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/bigdatainrealworld\" \/>\n<meta property=\"article:published_time\" content=\"2023-05-08T11:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-04-19T00:26:33+00:00\" \/>\n<meta name=\"author\" content=\"Big Data In Real World\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Big Data In Real World\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/\"},\"author\":{\"name\":\"Big Data In Real World\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\"},\"headline\":\"How to fix Kafka Broker may not be available on 127.0.0.1 error?\",\"datePublished\":\"2023-05-08T11:00:00+00:00\",\"dateModified\":\"2023-04-19T00:26:33+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/\"},\"wordCount\":434,\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"articleSection\":[\"Kafka\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/\",\"name\":\"How to fix Kafka Broker may not be available on 127.0.0.1 error? - Big Data In Real World\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\"},\"datePublished\":\"2023-05-08T11:00:00+00:00\",\"dateModified\":\"2023-04-19T00:26:33+00:00\",\"description\":\"Pretty much every Kafka developer has seen this error at least once. This post discuss the error, reason and solution for the same.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.bigdatainrealworld.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to fix Kafka Broker may not be available on 127.0.0.1 error?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"name\":\"Big Data In Real World\",\"description\":\"Learn Big Data from experts!\",\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\",\"name\":\"Big Data In Real World\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"contentUrl\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"width\":500,\"height\":500,\"caption\":\"Big Data In Real World\"},\"image\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/bigdatainrealworld\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\",\"name\":\"Big Data In Real World\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"caption\":\"Big Data In Real World\"},\"description\":\"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.\",\"sameAs\":[\"https:\/\/www.bigdatainrealworld.com\/\"],\"url\":\"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to fix Kafka Broker may not be available on 127.0.0.1 error? - Big Data In Real World","description":"Pretty much every Kafka developer has seen this error at least once. This post discuss the error, reason and solution for the same.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/","og_locale":"en_US","og_type":"article","og_title":"How to fix Kafka Broker may not be available on 127.0.0.1 error? - Big Data In Real World","og_description":"Pretty much every Kafka developer has seen this error at least once. This post discuss the error, reason and solution for the same.","og_url":"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/","og_site_name":"Big Data In Real World","article_publisher":"https:\/\/www.facebook.com\/bigdatainrealworld","article_published_time":"2023-05-08T11:00:00+00:00","article_modified_time":"2023-04-19T00:26:33+00:00","author":"Big Data In Real World","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Big Data In Real World","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/#article","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/"},"author":{"name":"Big Data In Real World","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67"},"headline":"How to fix Kafka Broker may not be available on 127.0.0.1 error?","datePublished":"2023-05-08T11:00:00+00:00","dateModified":"2023-04-19T00:26:33+00:00","mainEntityOfPage":{"@id":"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/"},"wordCount":434,"publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"articleSection":["Kafka"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/","url":"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/","name":"How to fix Kafka Broker may not be available on 127.0.0.1 error? - Big Data In Real World","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/#website"},"datePublished":"2023-05-08T11:00:00+00:00","dateModified":"2023-04-19T00:26:33+00:00","description":"Pretty much every Kafka developer has seen this error at least once. This post discuss the error, reason and solution for the same.","breadcrumb":{"@id":"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.bigdatainrealworld.com\/how-to-fix-kafka-broker-may-not-be-available-on-127-0-0-1-error\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.bigdatainrealworld.com\/"},{"@type":"ListItem","position":2,"name":"How to fix Kafka Broker may not be available on 127.0.0.1 error?"}]},{"@type":"WebSite","@id":"https:\/\/www.bigdatainrealworld.com\/#website","url":"https:\/\/www.bigdatainrealworld.com\/","name":"Big Data In Real World","description":"Learn Big Data from experts!","publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.bigdatainrealworld.com\/#organization","name":"Big Data In Real World","url":"https:\/\/www.bigdatainrealworld.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","contentUrl":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","width":500,"height":500,"caption":"Big Data In Real World"},"image":{"@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/bigdatainrealworld"]},{"@type":"Person","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67","name":"Big Data In Real World","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","caption":"Big Data In Real World"},"description":"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.","sameAs":["https:\/\/www.bigdatainrealworld.com\/"],"url":"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/"}]}},"_links":{"self":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/2270"}],"collection":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/comments?post=2270"}],"version-history":[{"count":1,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/2270\/revisions"}],"predecessor-version":[{"id":2271,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/2270\/revisions\/2271"}],"wp:attachment":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/media?parent=2270"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/categories?post=2270"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/tags?post=2270"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}