{"id":1695,"date":"2021-09-17T06:00:00","date_gmt":"2021-09-17T11:00:00","guid":{"rendered":"https:\/\/www.bigdatainrealworld.com\/?p=1695"},"modified":"2023-02-19T07:31:05","modified_gmt":"2023-02-19T13:31:05","slug":"how-does-spark-decide-stages-and-tasks-during-execution-of-a-job","status":"publish","type":"post","link":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/","title":{"rendered":"How does Spark decide stages and tasks during execution of a Job?"},"content":{"rendered":"<p>Let\u2019s see this with an example. Here is our series of instructions in our Spark code. Let\u2019s see how Spark decide on stages and tasks with the below set of instructions.<\/p>\n<ol>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">READ dataset_X<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">FILTER on dataset_X<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">MAP operation on dataset_X<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">READ dataset_Y<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">MAP operation on dataset_Y<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">JOIN dataset_X and dataset_Y<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">FILTER on joined dataset<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">SAVE the output<\/span><\/li>\n<\/ol>\n<p><script src=\"https:\/\/embed.lpcontent.net\/leadboxes\/current\/embed.js\" async=\"\" defer=\"defer\"><\/script><\/p>\n<p style=\"border: 1px; border-style: solid; border-color: #005cb9; background-color: #f1f5f9; padding: 1em;\">Do you like us to send you a 47 page Definitive guide on Spark join algorithms? ===&gt; <button style=\"background: #afbf00; border-color: #afbf00; border-radius: 20px; color: #ffffff; display: inline-block; vertical-align: middle; padding: 16px 32px; min-width: 192px; border: 1px solid #afbf00; font-size: 1rem; font-family: Helvetica, Arial, sans-serif; text-align: center; outline: 0; line-height: 1; cursor: pointer; -webkit-transition: background 0.3s, color 0.3s, border 0.3s; transition: background 0.3s, color 0.3s, border 0.3s; box-shadow: 0px 2px 5px rgba(0, 0, 0, 0.6);\" data-leadbox-popup=\"iRwJt3SrgaavPhqQ3dnrtg\" data-leadbox-domain=\"ccubecompany.lpages.co\">Send me the guide<\/button><\/p>\n<h2><span style=\"font-weight: 400;\">Stages<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Spark will create a stage for each dataset<\/span><\/p>\n<p><span style=\"font-weight: 400;\">All consecutive narrow transformations (for eg. FILTER, MAP etc.) will be grouped together inside the stage<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Spark will create a stage when it encounter a wide transformation (for eg. JOIN, reduceByKey etc.).\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">For the above set of instructions, Spark will create 3 stages &#8211;<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><strong>First stage<\/strong> &#8211; Instructions 1, 2 and 3<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><strong>Second stage<\/strong> &#8211; Instructions 4 and 5<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><strong>Third stage<\/strong> &#8211; Instructions 6, 7 and 8<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Tasks<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Spark creates a task to execute a set of instructions inside a stage.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Number of tasks equals the number of partitions in a dataset. <a href=\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/\">Check this for more details<\/a>.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Task execute all consecutive narrow transformations inside a stage &#8211; it is called pipelining.<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><strong>Task in first stage<\/strong> will execute instructions 1, 2 and 3<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><strong>Task in second stage<\/strong> will execute instructions 4 and 5<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><strong>Task in the third stage<\/strong> will execute instructions 6, 7 and 8.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Let\u2019s see this with an example. Here is our series of instructions in our Spark code. Let\u2019s see how Spark decide on stages and tasks with<span class=\"excerpt-hellip\"> [\u2026]<\/span><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[],"class_list":["post-1695","post","type-post","status-publish","format-standard","hentry","category-spark"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How does Spark decide stages and tasks during execution of a Job? - Big Data In Real World<\/title>\n<meta name=\"description\" content=\"Clear and crisp post to help you understand stages and tasks in Spark and how Spark decides them.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How does Spark decide stages and tasks during execution of a Job? - Big Data In Real World\" \/>\n<meta property=\"og:description\" content=\"Clear and crisp post to help you understand stages and tasks in Spark and how Spark decides them.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/\" \/>\n<meta property=\"og:site_name\" content=\"Big Data In Real World\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/bigdatainrealworld\" \/>\n<meta property=\"article:published_time\" content=\"2021-09-17T11:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-02-19T13:31:05+00:00\" \/>\n<meta name=\"author\" content=\"Big Data In Real World\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Big Data In Real World\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/\"},\"author\":{\"name\":\"Big Data In Real World\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\"},\"headline\":\"How does Spark decide stages and tasks during execution of a Job?\",\"datePublished\":\"2021-09-17T11:00:00+00:00\",\"dateModified\":\"2023-02-19T13:31:05+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/\"},\"wordCount\":234,\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"articleSection\":[\"Spark\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/\",\"name\":\"How does Spark decide stages and tasks during execution of a Job? - Big Data In Real World\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\"},\"datePublished\":\"2021-09-17T11:00:00+00:00\",\"dateModified\":\"2023-02-19T13:31:05+00:00\",\"description\":\"Clear and crisp post to help you understand stages and tasks in Spark and how Spark decides them.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.bigdatainrealworld.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How does Spark decide stages and tasks during execution of a Job?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"name\":\"Big Data In Real World\",\"description\":\"Learn Big Data from experts!\",\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\",\"name\":\"Big Data In Real World\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"contentUrl\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"width\":500,\"height\":500,\"caption\":\"Big Data In Real World\"},\"image\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/bigdatainrealworld\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\",\"name\":\"Big Data In Real World\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"caption\":\"Big Data In Real World\"},\"description\":\"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.\",\"sameAs\":[\"https:\/\/www.bigdatainrealworld.com\/\"],\"url\":\"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How does Spark decide stages and tasks during execution of a Job? - Big Data In Real World","description":"Clear and crisp post to help you understand stages and tasks in Spark and how Spark decides them.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/","og_locale":"en_US","og_type":"article","og_title":"How does Spark decide stages and tasks during execution of a Job? - Big Data In Real World","og_description":"Clear and crisp post to help you understand stages and tasks in Spark and how Spark decides them.","og_url":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/","og_site_name":"Big Data In Real World","article_publisher":"https:\/\/www.facebook.com\/bigdatainrealworld","article_published_time":"2021-09-17T11:00:00+00:00","article_modified_time":"2023-02-19T13:31:05+00:00","author":"Big Data In Real World","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Big Data In Real World","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/#article","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/"},"author":{"name":"Big Data In Real World","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67"},"headline":"How does Spark decide stages and tasks during execution of a Job?","datePublished":"2021-09-17T11:00:00+00:00","dateModified":"2023-02-19T13:31:05+00:00","mainEntityOfPage":{"@id":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/"},"wordCount":234,"publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"articleSection":["Spark"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/","url":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/","name":"How does Spark decide stages and tasks during execution of a Job? - Big Data In Real World","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/#website"},"datePublished":"2021-09-17T11:00:00+00:00","dateModified":"2023-02-19T13:31:05+00:00","description":"Clear and crisp post to help you understand stages and tasks in Spark and how Spark decides them.","breadcrumb":{"@id":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-stages-and-tasks-during-execution-of-a-job\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.bigdatainrealworld.com\/"},{"@type":"ListItem","position":2,"name":"How does Spark decide stages and tasks during execution of a Job?"}]},{"@type":"WebSite","@id":"https:\/\/www.bigdatainrealworld.com\/#website","url":"https:\/\/www.bigdatainrealworld.com\/","name":"Big Data In Real World","description":"Learn Big Data from experts!","publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.bigdatainrealworld.com\/#organization","name":"Big Data In Real World","url":"https:\/\/www.bigdatainrealworld.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","contentUrl":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","width":500,"height":500,"caption":"Big Data In Real World"},"image":{"@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/bigdatainrealworld"]},{"@type":"Person","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67","name":"Big Data In Real World","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","caption":"Big Data In Real World"},"description":"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.","sameAs":["https:\/\/www.bigdatainrealworld.com\/"],"url":"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/"}]}},"_links":{"self":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1695"}],"collection":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/comments?post=1695"}],"version-history":[{"count":3,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1695\/revisions"}],"predecessor-version":[{"id":2089,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1695\/revisions\/2089"}],"wp:attachment":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/media?parent=1695"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/categories?post=1695"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/tags?post=1695"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}