{"id":1699,"date":"2021-08-04T07:05:59","date_gmt":"2021-08-04T12:05:59","guid":{"rendered":"https:\/\/www.bigdatainrealworld.com\/?p=1699"},"modified":"2023-02-19T07:31:17","modified_gmt":"2023-02-19T13:31:17","slug":"how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel","status":"publish","type":"post","link":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/","title":{"rendered":"How does Spark decide the number of tasks and number of tasks to execute in parallel?"},"content":{"rendered":"<p>In this post we will see how Spark decides the number of tasks and number of tasks to execute in parallel in a job.<\/p>\n<p><span style=\"font-weight: 400;\">Let\u2019s see how Spark decides on the number of tasks with the below set of instructions.<\/span><\/p>\n<ol>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">READ dataset_X<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">FILTER on dataset_X<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">MAP operation on dataset_X<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">READ dataset_Y<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">MAP operation on dataset_Y<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">JOIN dataset_X and dataset_Y<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">FILTER on joined dataset<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">SAVE the output<\/span><\/li>\n<\/ol>\n<p><span style=\"font-weight: 400;\">Let\u2019s also assume dataset_Y has 10 partitions and dataset_Y has 5 partitions.<\/span><br \/>\n<script src=\"https:\/\/embed.lpcontent.net\/leadboxes\/current\/embed.js\" async=\"\" defer=\"defer\"><\/script><\/p>\n<p style=\"border: 1px; border-style: solid; border-color: #005cb9; background-color: #f1f5f9; padding: 1em;\">Do you like us to send you a 47 page Definitive guide on Spark join algorithms? ===&gt; <button style=\"background: #afbf00; border-color: #afbf00; border-radius: 20px; color: #ffffff; display: inline-block; vertical-align: middle; padding: 16px 32px; min-width: 192px; border: 1px solid #afbf00; font-size: 1rem; font-family: Helvetica, Arial, sans-serif; text-align: center; outline: 0; line-height: 1; cursor: pointer; -webkit-transition: background 0.3s, color 0.3s, border 0.3s; transition: background 0.3s, color 0.3s, border 0.3s; box-shadow: 0px 2px 5px rgba(0, 0, 0, 0.6);\" data-leadbox-popup=\"iRwJt3SrgaavPhqQ3dnrtg\" data-leadbox-domain=\"ccubecompany.lpages.co\">Send me the guide<\/button><\/p>\n<h2><span style=\"font-weight: 400;\">Stages and number of tasks per stage<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Spark will create 3 stages &#8211;<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><strong>First stage<\/strong> &#8211; Instructions 1, 2 and 3<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><strong>Second stage<\/strong> &#8211; Instructions 4 and 5<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><strong>Third stage<\/strong> &#8211; Instructions 6, 7 and 8<\/span><\/p>\n<h3><span style=\"font-weight: 400;\">Number of tasks in first stage&nbsp;<\/span><\/h3>\n<p><span style=\"font-weight: 400;\">First stage reads dataset_X and dataset_X has 10 partitions. So stage 1 will result in 10 tasks.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">If your dataset is very small, you might see Spark still creates 2 tasks and this is because Spark looks at the <\/span><span style=\"font-weight: 400;\">defaultMinPartitions<\/span><span style=\"font-weight: 400;\"> property and this property decides the minimum number of tasks Spark can create. The default for <\/span><span style=\"font-weight: 400;\">defaultMinPartitions<\/span><span style=\"font-weight: 400;\"> is 2.&nbsp;<\/span><\/p>\n<h3><span style=\"font-weight: 400;\">Number of tasks in second stage&nbsp;<\/span><\/h3>\n<p><span style=\"font-weight: 400;\">Second stage reads dataset_Y and dataset_Y has 5 partitions. So stage 2 will result in 5 tasks.<\/span><\/p>\n<h3><span style=\"font-weight: 400;\">Number of tasks in third stage&nbsp;<\/span><\/h3>\n<p><span style=\"font-weight: 400;\">Third stage executes a JOIN and JOIN operation triggers a wide transformation and wide transformation will result in a shuffle. Spark optimizer tries to pick the \u201cright\u201d number of partitions during a shuffle but most often you will see Spark creates 200 tasks for stages executing wide transformation operations like JOIN, GROUP BY etc.<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><span class=\"lang:default decode:true crayon-inline \">spark.sql.shuffle.partitions<\/span>&nbsp; property controls the number of partitions during a shuffle and the default value of this property is 200.&nbsp;<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Change the value of <span class=\"lang:default decode:true crayon-inline \">spark.sql.shuffle.partitions<\/span>&nbsp; to change the number of partitions during a shuffle.<\/span><\/p>\n<pre class=\"lang:default decode:true \">sqlContext.setConf(\"spark.sql.shuffle.partitions\", \"8\u201d)<\/pre>\n<p>&nbsp;<\/p>\n<h2><span style=\"font-weight: 400;\">Number of tasks execution in parallel<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Number of CPU cores available for an executor determines the number of tasks that can be executed in parallel for an application for any given time.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Let\u2019s say, you have 5 executors available for your application. Each executor is assigned 10 CPU cores.&nbsp;<\/span><\/p>\n<p><span style=\"font-weight: 400;\">5 executors and 10 CPU cores per executor = 50 CPU cores available in total.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">With the above setup, Spark can execute a maximum of 50 tasks in parallel at any given time.<\/span><\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this post we will see how Spark decides the number of tasks and number of tasks to execute in parallel in a job. Let\u2019s see<span class=\"excerpt-hellip\"> [\u2026]<\/span><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[],"class_list":["post-1699","post","type-post","status-publish","format-standard","hentry","category-spark"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How does Spark decide the number of tasks and number of tasks to execute in parallel? - Big Data In Real World<\/title>\n<meta name=\"description\" content=\"This is THE post if you are confused about tasks, no of tasks and parallel execution of tasks in Spark.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How does Spark decide the number of tasks and number of tasks to execute in parallel? - Big Data In Real World\" \/>\n<meta property=\"og:description\" content=\"This is THE post if you are confused about tasks, no of tasks and parallel execution of tasks in Spark.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/\" \/>\n<meta property=\"og:site_name\" content=\"Big Data In Real World\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/bigdatainrealworld\" \/>\n<meta property=\"article:published_time\" content=\"2021-08-04T12:05:59+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-02-19T13:31:17+00:00\" \/>\n<meta name=\"author\" content=\"Big Data In Real World\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Big Data In Real World\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/\"},\"author\":{\"name\":\"Big Data In Real World\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\"},\"headline\":\"How does Spark decide the number of tasks and number of tasks to execute in parallel?\",\"datePublished\":\"2021-08-04T12:05:59+00:00\",\"dateModified\":\"2023-02-19T13:31:17+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/\"},\"wordCount\":417,\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"articleSection\":[\"Spark\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/\",\"name\":\"How does Spark decide the number of tasks and number of tasks to execute in parallel? - Big Data In Real World\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\"},\"datePublished\":\"2021-08-04T12:05:59+00:00\",\"dateModified\":\"2023-02-19T13:31:17+00:00\",\"description\":\"This is THE post if you are confused about tasks, no of tasks and parallel execution of tasks in Spark.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.bigdatainrealworld.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How does Spark decide the number of tasks and number of tasks to execute in parallel?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"name\":\"Big Data In Real World\",\"description\":\"Learn Big Data from experts!\",\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\",\"name\":\"Big Data In Real World\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"contentUrl\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"width\":500,\"height\":500,\"caption\":\"Big Data In Real World\"},\"image\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/bigdatainrealworld\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\",\"name\":\"Big Data In Real World\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"caption\":\"Big Data In Real World\"},\"description\":\"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.\",\"sameAs\":[\"https:\/\/www.bigdatainrealworld.com\/\"],\"url\":\"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How does Spark decide the number of tasks and number of tasks to execute in parallel? - Big Data In Real World","description":"This is THE post if you are confused about tasks, no of tasks and parallel execution of tasks in Spark.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/","og_locale":"en_US","og_type":"article","og_title":"How does Spark decide the number of tasks and number of tasks to execute in parallel? - Big Data In Real World","og_description":"This is THE post if you are confused about tasks, no of tasks and parallel execution of tasks in Spark.","og_url":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/","og_site_name":"Big Data In Real World","article_publisher":"https:\/\/www.facebook.com\/bigdatainrealworld","article_published_time":"2021-08-04T12:05:59+00:00","article_modified_time":"2023-02-19T13:31:17+00:00","author":"Big Data In Real World","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Big Data In Real World","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/#article","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/"},"author":{"name":"Big Data In Real World","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67"},"headline":"How does Spark decide the number of tasks and number of tasks to execute in parallel?","datePublished":"2021-08-04T12:05:59+00:00","dateModified":"2023-02-19T13:31:17+00:00","mainEntityOfPage":{"@id":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/"},"wordCount":417,"publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"articleSection":["Spark"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/","url":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/","name":"How does Spark decide the number of tasks and number of tasks to execute in parallel? - Big Data In Real World","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/#website"},"datePublished":"2021-08-04T12:05:59+00:00","dateModified":"2023-02-19T13:31:17+00:00","description":"This is THE post if you are confused about tasks, no of tasks and parallel execution of tasks in Spark.","breadcrumb":{"@id":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.bigdatainrealworld.com\/"},{"@type":"ListItem","position":2,"name":"How does Spark decide the number of tasks and number of tasks to execute in parallel?"}]},{"@type":"WebSite","@id":"https:\/\/www.bigdatainrealworld.com\/#website","url":"https:\/\/www.bigdatainrealworld.com\/","name":"Big Data In Real World","description":"Learn Big Data from experts!","publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.bigdatainrealworld.com\/#organization","name":"Big Data In Real World","url":"https:\/\/www.bigdatainrealworld.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","contentUrl":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","width":500,"height":500,"caption":"Big Data In Real World"},"image":{"@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/bigdatainrealworld"]},{"@type":"Person","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67","name":"Big Data In Real World","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","caption":"Big Data In Real World"},"description":"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.","sameAs":["https:\/\/www.bigdatainrealworld.com\/"],"url":"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/"}]}},"_links":{"self":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1699"}],"collection":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/comments?post=1699"}],"version-history":[{"count":3,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1699\/revisions"}],"predecessor-version":[{"id":2071,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1699\/revisions\/2071"}],"wp:attachment":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/media?parent=1699"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/categories?post=1699"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/tags?post=1699"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}