{"id":1693,"date":"2021-01-25T06:00:43","date_gmt":"2021-01-25T12:00:43","guid":{"rendered":"https:\/\/www.bigdatainrealworld.com\/?p=1693"},"modified":"2023-02-19T07:31:58","modified_gmt":"2023-02-19T13:31:58","slug":"how-to-control-log-settings-in-spark-and-stop-info-messages","status":"publish","type":"post","link":"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/","title":{"rendered":"How to control log settings in Spark and stop INFO messages?"},"content":{"rendered":"<p>Spark spits out a lot of INFO messages in the logs. These messages stand in the way when you are troubleshooting an error and looking for error messages in the logs.<\/p>\n<p><span style=\"font-weight: 400;\">Here is an example of log messages that you might see on the console or in the log.<\/span><\/p>\n<pre class=\"lang:default decode:true \">20\/10\/10 10:12:03 INFO SparkEnv: Registering BlockManagerMaster\r\n20\/10\/10 10:12:03 INFO DiskBlockManager: Created local directory at \/tmp\/spark-local-201201010101203-xyik\r\n20\/10\/10 10:12:03 INFO MemoryStore: MemoryStore started with capacity 0.0 B.\r\n20\/10\/10 10:12:03 INFO ConnectionManager: Bound socket to port 44728 with id = ConnectionManagerId(10.1.100.12,44728)\r\n20\/10\/10 10:12:03 INFO BlockManagerMaster: Trying to register BlockManager\r\n20\/10\/10 10:12:03 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager 10.1.100.12:44728 with 100.0 B RAM\r\n20\/10\/10 10:12:03 INFO BlockManagerMaster: Registered BlockManager\r\n20\/10\/10 10:12:03 INFO HttpServer: Starting HTTP Server\r\n20\/10\/10 10:12:03 INFO HttpBroadcast: Broadcast server started<\/pre>\n<p><script src=\"https:\/\/embed.lpcontent.net\/leadboxes\/current\/embed.js\" async=\"\" defer=\"defer\"><\/script><\/p>\n<p style=\"border: 1px; border-style: solid; border-color: #005cb9; background-color: #f1f5f9; padding: 1em;\">Do you like us to send you a 47 page Definitive guide on Spark join algorithms? ===&gt; <button style=\"background: #afbf00; border-color: #afbf00; border-radius: 20px; color: #ffffff; display: inline-block; vertical-align: middle; padding: 16px 32px; min-width: 192px; border: 1px solid #afbf00; font-size: 1rem; font-family: Helvetica, Arial, sans-serif; text-align: center; outline: 0; line-height: 1; cursor: pointer; -webkit-transition: background 0.3s, color 0.3s, border 0.3s; transition: background 0.3s, color 0.3s, border 0.3s; box-shadow: 0px 2px 5px rgba(0, 0, 0, 0.6);\" data-leadbox-popup=\"iRwJt3SrgaavPhqQ3dnrtg\" data-leadbox-domain=\"ccubecompany.lpages.co\">Send me the guide<\/button><\/p>\n<h2><span style=\"font-weight: 400;\">Changing the log level in log4j.properties<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">log4j.properties controls the log related configuration settings. You can find this file under $SPARK_HOME\/conf\/log4j.properties<\/span><\/p>\n<p><span style=\"font-weight: 400;\">There are several log levels in log4j &#8211;<span class=\"lang:default decode:true crayon-inline\">OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE, ALL<\/span>\u00a0<\/span><span style=\"font-weight: 400;\">. FATAL will list only FATAL messages. ERROR level will list only ERROR and FATAL messages. ALL will list all log messages.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">You will see <span class=\"lang:default decode:true crayon-inline \">log4j.rootCategory=INFO<\/span>\u00a0, console in <\/span><span style=\"font-weight: 400;\">log4j.properties files for Spark. This means you will see all log messages from INFO, WARN, ERROR and FATAL. If you want to only see ERROR and FATAL messages change this property to <span class=\"lang:default decode:true crayon-inline \">log4j.rootCategory=ERROR<\/span>\u00a0<\/span><span style=\"font-weight: 400;\">, console.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Changing the log level in code<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Changing the log level in log4j.properties file will affect all applications. If you want to control the logging in specific applications then you can override the properties in your code.<\/span><\/p>\n<pre class=\"lang:default decode:true \">spark.sparkContext.setLogLevel(\"ERROR\")<\/pre>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Spark spits out a lot of INFO messages in the logs. These messages stand in the way when you are troubleshooting an error and looking for<span class=\"excerpt-hellip\"> [\u2026]<\/span><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[],"class_list":["post-1693","post","type-post","status-publish","format-standard","hentry","category-spark"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How to control log settings in Spark and stop INFO messages? - Big Data In Real World<\/title>\n<meta name=\"description\" content=\"Learn how to control log messages in this post.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to control log settings in Spark and stop INFO messages? - Big Data In Real World\" \/>\n<meta property=\"og:description\" content=\"Learn how to control log messages in this post.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/\" \/>\n<meta property=\"og:site_name\" content=\"Big Data In Real World\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/bigdatainrealworld\" \/>\n<meta property=\"article:published_time\" content=\"2021-01-25T12:00:43+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-02-19T13:31:58+00:00\" \/>\n<meta name=\"author\" content=\"Big Data In Real World\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Big Data In Real World\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/\"},\"author\":{\"name\":\"Big Data In Real World\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\"},\"headline\":\"How to control log settings in Spark and stop INFO messages?\",\"datePublished\":\"2021-01-25T12:00:43+00:00\",\"dateModified\":\"2023-02-19T13:31:58+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/\"},\"wordCount\":233,\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"articleSection\":[\"Spark\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/\",\"name\":\"How to control log settings in Spark and stop INFO messages? - Big Data In Real World\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\"},\"datePublished\":\"2021-01-25T12:00:43+00:00\",\"dateModified\":\"2023-02-19T13:31:58+00:00\",\"description\":\"Learn how to control log messages in this post.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.bigdatainrealworld.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to control log settings in Spark and stop INFO messages?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"name\":\"Big Data In Real World\",\"description\":\"Learn Big Data from experts!\",\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\",\"name\":\"Big Data In Real World\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"contentUrl\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"width\":500,\"height\":500,\"caption\":\"Big Data In Real World\"},\"image\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/bigdatainrealworld\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\",\"name\":\"Big Data In Real World\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"caption\":\"Big Data In Real World\"},\"description\":\"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.\",\"sameAs\":[\"https:\/\/www.bigdatainrealworld.com\/\"],\"url\":\"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to control log settings in Spark and stop INFO messages? - Big Data In Real World","description":"Learn how to control log messages in this post.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/","og_locale":"en_US","og_type":"article","og_title":"How to control log settings in Spark and stop INFO messages? - Big Data In Real World","og_description":"Learn how to control log messages in this post.","og_url":"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/","og_site_name":"Big Data In Real World","article_publisher":"https:\/\/www.facebook.com\/bigdatainrealworld","article_published_time":"2021-01-25T12:00:43+00:00","article_modified_time":"2023-02-19T13:31:58+00:00","author":"Big Data In Real World","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Big Data In Real World","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/#article","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/"},"author":{"name":"Big Data In Real World","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67"},"headline":"How to control log settings in Spark and stop INFO messages?","datePublished":"2021-01-25T12:00:43+00:00","dateModified":"2023-02-19T13:31:58+00:00","mainEntityOfPage":{"@id":"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/"},"wordCount":233,"publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"articleSection":["Spark"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/","url":"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/","name":"How to control log settings in Spark and stop INFO messages? - Big Data In Real World","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/#website"},"datePublished":"2021-01-25T12:00:43+00:00","dateModified":"2023-02-19T13:31:58+00:00","description":"Learn how to control log messages in this post.","breadcrumb":{"@id":"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.bigdatainrealworld.com\/how-to-control-log-settings-in-spark-and-stop-info-messages\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.bigdatainrealworld.com\/"},{"@type":"ListItem","position":2,"name":"How to control log settings in Spark and stop INFO messages?"}]},{"@type":"WebSite","@id":"https:\/\/www.bigdatainrealworld.com\/#website","url":"https:\/\/www.bigdatainrealworld.com\/","name":"Big Data In Real World","description":"Learn Big Data from experts!","publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.bigdatainrealworld.com\/#organization","name":"Big Data In Real World","url":"https:\/\/www.bigdatainrealworld.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","contentUrl":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","width":500,"height":500,"caption":"Big Data In Real World"},"image":{"@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/bigdatainrealworld"]},{"@type":"Person","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67","name":"Big Data In Real World","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","caption":"Big Data In Real World"},"description":"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.","sameAs":["https:\/\/www.bigdatainrealworld.com\/"],"url":"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/"}]}},"_links":{"self":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1693"}],"collection":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/comments?post=1693"}],"version-history":[{"count":2,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1693\/revisions"}],"predecessor-version":[{"id":1959,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1693\/revisions\/1959"}],"wp:attachment":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/media?parent=1693"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/categories?post=1693"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/tags?post=1693"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}