{"id":1740,"date":"2021-09-06T06:00:00","date_gmt":"2021-09-06T11:00:00","guid":{"rendered":"https:\/\/www.bigdatainrealworld.com\/?p=1740"},"modified":"2023-02-19T07:31:06","modified_gmt":"2023-02-19T13:31:06","slug":"what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui","status":"publish","type":"post","link":"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/","title":{"rendered":"What do the numbers on the progress bar mean in Spark shell or Spark UI?"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">We are sure you have seen the below progress bar before in either Spark shell or while refreshing the Spark UI as your Spark job execution is in progress.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">You probably wondered what these numbers mean.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">[Stage 2 : ===========&gt; (45 + 5) \/ 150]<\/span><br \/>\n<script src=\"https:\/\/embed.lpcontent.net\/leadboxes\/current\/embed.js\" async=\"\" defer=\"defer\"><\/script><\/p>\n<p style=\"border: 1px; border-style: solid; border-color: #005cb9; background-color: #f1f5f9; padding: 1em;\">Do you like us to send you a 47 page Definitive guide on Spark join algorithms? ===&gt; <button style=\"background: #afbf00; border-color: #afbf00; border-radius: 20px; color: #ffffff; display: inline-block; vertical-align: middle; padding: 16px 32px; min-width: 192px; border: 1px solid #afbf00; font-size: 1rem; font-family: Helvetica, Arial, sans-serif; text-align: center; outline: 0; line-height: 1; cursor: pointer; -webkit-transition: background 0.3s, color 0.3s, border 0.3s; transition: background 0.3s, color 0.3s, border 0.3s; box-shadow: 0px 2px 5px rgba(0, 0, 0, 0.6);\" data-leadbox-popup=\"iRwJt3SrgaavPhqQ3dnrtg\" data-leadbox-domain=\"ccubecompany.lpages.co\">Send me the guide<\/button><\/p>\n<h2><span style=\"font-weight: 400;\">Spark progress bar<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Stage 2 &#8211; quite simple. Indicates the current stage in execution<\/span><\/p>\n<p><span style=\"font-weight: 400;\">150 &#8211; Total number of tasks in Stage 2<\/span><\/p>\n<p><span style=\"font-weight: 400;\">45 &#8211; Number of tasks completed in Stage 2<\/span><\/p>\n<p><span style=\"font-weight: 400;\">5 &#8211; Number of tasks currently in execution in Stage 2<\/span><\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>We are sure you have seen the below progress bar before in either Spark shell or while refreshing the Spark UI as your Spark job execution<span class=\"excerpt-hellip\"> [\u2026]<\/span><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[],"class_list":["post-1740","post","type-post","status-publish","format-standard","hentry","category-spark"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>What do the numbers on the progress bar mean in Spark shell or Spark UI? - Big Data In Real World<\/title>\n<meta name=\"description\" content=\"A quick 1 minute explanation to understand the numbers on the progress bar during Spark job execution.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What do the numbers on the progress bar mean in Spark shell or Spark UI? - Big Data In Real World\" \/>\n<meta property=\"og:description\" content=\"A quick 1 minute explanation to understand the numbers on the progress bar during Spark job execution.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/\" \/>\n<meta property=\"og:site_name\" content=\"Big Data In Real World\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/bigdatainrealworld\" \/>\n<meta property=\"article:published_time\" content=\"2021-09-06T11:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-02-19T13:31:06+00:00\" \/>\n<meta name=\"author\" content=\"Big Data In Real World\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Big Data In Real World\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/\"},\"author\":{\"name\":\"Big Data In Real World\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\"},\"headline\":\"What do the numbers on the progress bar mean in Spark shell or Spark UI?\",\"datePublished\":\"2021-09-06T11:00:00+00:00\",\"dateModified\":\"2023-02-19T13:31:06+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/\"},\"wordCount\":110,\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"articleSection\":[\"Spark\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/\",\"name\":\"What do the numbers on the progress bar mean in Spark shell or Spark UI? - Big Data In Real World\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\"},\"datePublished\":\"2021-09-06T11:00:00+00:00\",\"dateModified\":\"2023-02-19T13:31:06+00:00\",\"description\":\"A quick 1 minute explanation to understand the numbers on the progress bar during Spark job execution.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.bigdatainrealworld.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What do the numbers on the progress bar mean in Spark shell or Spark UI?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"name\":\"Big Data In Real World\",\"description\":\"Learn Big Data from experts!\",\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\",\"name\":\"Big Data In Real World\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"contentUrl\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"width\":500,\"height\":500,\"caption\":\"Big Data In Real World\"},\"image\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/bigdatainrealworld\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\",\"name\":\"Big Data In Real World\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"caption\":\"Big Data In Real World\"},\"description\":\"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.\",\"sameAs\":[\"https:\/\/www.bigdatainrealworld.com\/\"],\"url\":\"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What do the numbers on the progress bar mean in Spark shell or Spark UI? - Big Data In Real World","description":"A quick 1 minute explanation to understand the numbers on the progress bar during Spark job execution.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/","og_locale":"en_US","og_type":"article","og_title":"What do the numbers on the progress bar mean in Spark shell or Spark UI? - Big Data In Real World","og_description":"A quick 1 minute explanation to understand the numbers on the progress bar during Spark job execution.","og_url":"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/","og_site_name":"Big Data In Real World","article_publisher":"https:\/\/www.facebook.com\/bigdatainrealworld","article_published_time":"2021-09-06T11:00:00+00:00","article_modified_time":"2023-02-19T13:31:06+00:00","author":"Big Data In Real World","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Big Data In Real World","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/#article","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/"},"author":{"name":"Big Data In Real World","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67"},"headline":"What do the numbers on the progress bar mean in Spark shell or Spark UI?","datePublished":"2021-09-06T11:00:00+00:00","dateModified":"2023-02-19T13:31:06+00:00","mainEntityOfPage":{"@id":"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/"},"wordCount":110,"publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"articleSection":["Spark"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/","url":"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/","name":"What do the numbers on the progress bar mean in Spark shell or Spark UI? - Big Data In Real World","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/#website"},"datePublished":"2021-09-06T11:00:00+00:00","dateModified":"2023-02-19T13:31:06+00:00","description":"A quick 1 minute explanation to understand the numbers on the progress bar during Spark job execution.","breadcrumb":{"@id":"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.bigdatainrealworld.com\/what-do-the-numbers-on-the-progress-bar-mean-in-spark-shell-or-spark-ui\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.bigdatainrealworld.com\/"},{"@type":"ListItem","position":2,"name":"What do the numbers on the progress bar mean in Spark shell or Spark UI?"}]},{"@type":"WebSite","@id":"https:\/\/www.bigdatainrealworld.com\/#website","url":"https:\/\/www.bigdatainrealworld.com\/","name":"Big Data In Real World","description":"Learn Big Data from experts!","publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.bigdatainrealworld.com\/#organization","name":"Big Data In Real World","url":"https:\/\/www.bigdatainrealworld.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","contentUrl":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","width":500,"height":500,"caption":"Big Data In Real World"},"image":{"@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/bigdatainrealworld"]},{"@type":"Person","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67","name":"Big Data In Real World","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","caption":"Big Data In Real World"},"description":"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.","sameAs":["https:\/\/www.bigdatainrealworld.com\/"],"url":"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/"}]}},"_links":{"self":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1740"}],"collection":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/comments?post=1740"}],"version-history":[{"count":2,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1740\/revisions"}],"predecessor-version":[{"id":1980,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1740\/revisions\/1980"}],"wp:attachment":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/media?parent=1740"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/categories?post=1740"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/tags?post=1740"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}