{"id":1697,"date":"2021-09-24T06:00:00","date_gmt":"2021-09-24T11:00:00","guid":{"rendered":"https:\/\/www.bigdatainrealworld.com\/?p=1697"},"modified":"2023-02-19T07:31:05","modified_gmt":"2023-02-19T13:31:05","slug":"why-do-i-see-200-tasks-in-spark-execution","status":"publish","type":"post","link":"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/","title":{"rendered":"Why do I see 200 tasks in Spark execution?"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">It is quite common to see 200 tasks in one of your stages and more specifically at a stage which requires wide transformation. The reason for this is, wide transformations in Spark requires a shuffle. Operations like join, group by etc. are wide transform operations and they trigger a shuffle.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">By default, Spark creates 200 partitions whenever there is a need for shuffle. Each partition will be processed by a task. So, you will end up with 200 tasks during execution.<\/span><br \/>\n<script src=\"https:\/\/embed.lpcontent.net\/leadboxes\/current\/embed.js\" async=\"\" defer=\"defer\"><\/script><\/p>\n<p style=\"border: 1px; border-style: solid; border-color: #005cb9; background-color: #f1f5f9; padding: 1em;\">Do you like us to send you a 47 page Definitive guide on Spark join algorithms? ===&gt; <button style=\"background: #afbf00; border-color: #afbf00; border-radius: 20px; color: #ffffff; display: inline-block; vertical-align: middle; padding: 16px 32px; min-width: 192px; border: 1px solid #afbf00; font-size: 1rem; font-family: Helvetica, Arial, sans-serif; text-align: center; outline: 0; line-height: 1; cursor: pointer; -webkit-transition: background 0.3s, color 0.3s, border 0.3s; transition: background 0.3s, color 0.3s, border 0.3s; box-shadow: 0px 2px 5px rgba(0, 0, 0, 0.6);\" data-leadbox-popup=\"iRwJt3SrgaavPhqQ3dnrtg\" data-leadbox-domain=\"ccubecompany.lpages.co\">Send me the guide<\/button><\/p>\n<h2><span style=\"font-weight: 400;\">How to change the default 200 tasks?<\/span><\/h2>\n<p><span style=\"font-weight: 400;\"><span class=\"lang:default decode:true crayon-inline \">spark.sql.shuffle.partitions<\/span>&nbsp; property controls the number of partitions during a shuffle and the default value of this property is 200.&nbsp;<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Change the value of <span class=\"lang:default decode:true crayon-inline \">spark.sql.shuffle.partitions<\/span>&nbsp; to change the number of partitions during a shuffle.<\/span><\/p>\n<pre class=\"lang:default decode:true \">sqlContext.setConf(\"spark.sql.shuffle.partitions\", \"4\u201d)<\/pre>\n<h2><span style=\"font-weight: 400;\">Should you change the default?<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Short answer is &#8211; it depends.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">200 partitions could be a lot when your data volume is small. Why? Because, each partition is processed by a task and each task is processing a small amount of data and this could result in performance issues. Very simply, you can decrease the number of partitions which would result in fewer tasks and that would result in better performance.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">If you are writing the data after a shuffle and if your data volume is small, you would result with 200 small files by default and this is another reason why you might want to consider changing the default to a smaller number.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">200 partitions could be low when the amount of data involved in shuffle is huge. If a task is processing a lot of data you could see out of memory exceptions or slower tasks executions. This could be rectified by increasing the number of partitions there by increasing the number tasks which in turn allow each task to process a manageable amount of data.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>It is quite common to see 200 tasks in one of your stages and more specifically at a stage which requires wide transformation. The reason for<span class=\"excerpt-hellip\"> [\u2026]<\/span><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[],"class_list":["post-1697","post","type-post","status-publish","format-standard","hentry","category-spark"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Why do I see 200 tasks in Spark execution? - Big Data In Real World<\/title>\n<meta name=\"description\" content=\"Confused on why do you see 200 tasks in Spark? Check this post for the answer.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Why do I see 200 tasks in Spark execution? - Big Data In Real World\" \/>\n<meta property=\"og:description\" content=\"Confused on why do you see 200 tasks in Spark? Check this post for the answer.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/\" \/>\n<meta property=\"og:site_name\" content=\"Big Data In Real World\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/bigdatainrealworld\" \/>\n<meta property=\"article:published_time\" content=\"2021-09-24T11:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-02-19T13:31:05+00:00\" \/>\n<meta name=\"author\" content=\"Big Data In Real World\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Big Data In Real World\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/\"},\"author\":{\"name\":\"Big Data In Real World\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\"},\"headline\":\"Why do I see 200 tasks in Spark execution?\",\"datePublished\":\"2021-09-24T11:00:00+00:00\",\"dateModified\":\"2023-02-19T13:31:05+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/\"},\"wordCount\":328,\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"articleSection\":[\"Spark\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/\",\"name\":\"Why do I see 200 tasks in Spark execution? - Big Data In Real World\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\"},\"datePublished\":\"2021-09-24T11:00:00+00:00\",\"dateModified\":\"2023-02-19T13:31:05+00:00\",\"description\":\"Confused on why do you see 200 tasks in Spark? Check this post for the answer.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.bigdatainrealworld.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Why do I see 200 tasks in Spark execution?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"name\":\"Big Data In Real World\",\"description\":\"Learn Big Data from experts!\",\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\",\"name\":\"Big Data In Real World\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"contentUrl\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"width\":500,\"height\":500,\"caption\":\"Big Data In Real World\"},\"image\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/bigdatainrealworld\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\",\"name\":\"Big Data In Real World\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"caption\":\"Big Data In Real World\"},\"description\":\"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.\",\"sameAs\":[\"https:\/\/www.bigdatainrealworld.com\/\"],\"url\":\"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Why do I see 200 tasks in Spark execution? - Big Data In Real World","description":"Confused on why do you see 200 tasks in Spark? Check this post for the answer.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/","og_locale":"en_US","og_type":"article","og_title":"Why do I see 200 tasks in Spark execution? - Big Data In Real World","og_description":"Confused on why do you see 200 tasks in Spark? Check this post for the answer.","og_url":"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/","og_site_name":"Big Data In Real World","article_publisher":"https:\/\/www.facebook.com\/bigdatainrealworld","article_published_time":"2021-09-24T11:00:00+00:00","article_modified_time":"2023-02-19T13:31:05+00:00","author":"Big Data In Real World","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Big Data In Real World","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/#article","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/"},"author":{"name":"Big Data In Real World","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67"},"headline":"Why do I see 200 tasks in Spark execution?","datePublished":"2021-09-24T11:00:00+00:00","dateModified":"2023-02-19T13:31:05+00:00","mainEntityOfPage":{"@id":"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/"},"wordCount":328,"publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"articleSection":["Spark"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/","url":"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/","name":"Why do I see 200 tasks in Spark execution? - Big Data In Real World","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/#website"},"datePublished":"2021-09-24T11:00:00+00:00","dateModified":"2023-02-19T13:31:05+00:00","description":"Confused on why do you see 200 tasks in Spark? Check this post for the answer.","breadcrumb":{"@id":"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.bigdatainrealworld.com\/why-do-i-see-200-tasks-in-spark-execution\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.bigdatainrealworld.com\/"},{"@type":"ListItem","position":2,"name":"Why do I see 200 tasks in Spark execution?"}]},{"@type":"WebSite","@id":"https:\/\/www.bigdatainrealworld.com\/#website","url":"https:\/\/www.bigdatainrealworld.com\/","name":"Big Data In Real World","description":"Learn Big Data from experts!","publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.bigdatainrealworld.com\/#organization","name":"Big Data In Real World","url":"https:\/\/www.bigdatainrealworld.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","contentUrl":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","width":500,"height":500,"caption":"Big Data In Real World"},"image":{"@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/bigdatainrealworld"]},{"@type":"Person","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67","name":"Big Data In Real World","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","caption":"Big Data In Real World"},"description":"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.","sameAs":["https:\/\/www.bigdatainrealworld.com\/"],"url":"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/"}]}},"_links":{"self":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1697"}],"collection":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/comments?post=1697"}],"version-history":[{"count":3,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1697\/revisions"}],"predecessor-version":[{"id":2092,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1697\/revisions\/2092"}],"wp:attachment":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/media?parent=1697"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/categories?post=1697"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/tags?post=1697"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}