{"id":2307,"date":"2023-07-31T06:00:00","date_gmt":"2023-07-31T11:00:00","guid":{"rendered":"https:\/\/www.bigdatainrealworld.com\/?p=2307"},"modified":"2023-07-18T06:49:54","modified_gmt":"2023-07-18T11:49:54","slug":"what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark","status":"publish","type":"post","link":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/","title":{"rendered":"What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism in Spark?"},"content":{"rendered":"\n<p>Both spark.sql.shuffle.partitions and spark.default.parallelism control the number of tasks that get executed at runtime there by controlling the distribution and parallelism. Which means both properties have a direct effect on performance.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">spark.sql.shuffle.partitions<\/h2>\n\n\n\n<p>spark.sql.shuffle.partitions configures the number of partitions that are used when shuffling data for joins or aggregations. The default for this this property is 200.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">spark.default.parallelism<\/h2>\n\n\n\n<p>spark.default.parallelism is the default number of partitions in RDDs returned by transformations like join, reduceByKey, and parallelize when not set explicitly by the user.&nbsp;<\/p>\n\n\n\n<p>spark.default.parallelism works only with RDDs and is ignored when working with DataFrames.<\/p>\n\n\n\n<p>Default value depend on your deployment &#8211;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Local mode: number of cores on the local machine<\/li>\n\n\n\n<li>Mesos fine grained mode: 8<\/li>\n\n\n\n<li>Others: total number of cores on all executor nodes or 2, whichever is larger<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Both spark.sql.shuffle.partitions and spark.default.parallelism control the number of tasks that get executed at runtime there by controlling the distribution and parallelism. Which means both properties have<span class=\"excerpt-hellip\"> [\u2026]<\/span><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[],"class_list":["post-2307","post","type-post","status-publish","format-standard","hentry","category-spark"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism in Spark? - Big Data In Real World<\/title>\n<meta name=\"description\" content=\"Both spark.sql.shuffle.partitions and spark.default.parallelism have an impact on performance and this post explains the difference between the two.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism in Spark? - Big Data In Real World\" \/>\n<meta property=\"og:description\" content=\"Both spark.sql.shuffle.partitions and spark.default.parallelism have an impact on performance and this post explains the difference between the two.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/\" \/>\n<meta property=\"og:site_name\" content=\"Big Data In Real World\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/bigdatainrealworld\" \/>\n<meta property=\"article:published_time\" content=\"2023-07-31T11:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-07-18T11:49:54+00:00\" \/>\n<meta name=\"author\" content=\"Big Data In Real World\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Big Data In Real World\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/\"},\"author\":{\"name\":\"Big Data In Real World\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\"},\"headline\":\"What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism in Spark?\",\"datePublished\":\"2023-07-31T11:00:00+00:00\",\"dateModified\":\"2023-07-18T11:49:54+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/\"},\"wordCount\":158,\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"articleSection\":[\"Spark\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/\",\"name\":\"What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism in Spark? - Big Data In Real World\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\"},\"datePublished\":\"2023-07-31T11:00:00+00:00\",\"dateModified\":\"2023-07-18T11:49:54+00:00\",\"description\":\"Both spark.sql.shuffle.partitions and spark.default.parallelism have an impact on performance and this post explains the difference between the two.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.bigdatainrealworld.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism in Spark?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"name\":\"Big Data In Real World\",\"description\":\"Learn Big Data from experts!\",\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\",\"name\":\"Big Data In Real World\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"contentUrl\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"width\":500,\"height\":500,\"caption\":\"Big Data In Real World\"},\"image\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/bigdatainrealworld\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\",\"name\":\"Big Data In Real World\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"caption\":\"Big Data In Real World\"},\"description\":\"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.\",\"sameAs\":[\"https:\/\/www.bigdatainrealworld.com\/\"],\"url\":\"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism in Spark? - Big Data In Real World","description":"Both spark.sql.shuffle.partitions and spark.default.parallelism have an impact on performance and this post explains the difference between the two.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/","og_locale":"en_US","og_type":"article","og_title":"What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism in Spark? - Big Data In Real World","og_description":"Both spark.sql.shuffle.partitions and spark.default.parallelism have an impact on performance and this post explains the difference between the two.","og_url":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/","og_site_name":"Big Data In Real World","article_publisher":"https:\/\/www.facebook.com\/bigdatainrealworld","article_published_time":"2023-07-31T11:00:00+00:00","article_modified_time":"2023-07-18T11:49:54+00:00","author":"Big Data In Real World","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Big Data In Real World","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/#article","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/"},"author":{"name":"Big Data In Real World","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67"},"headline":"What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism in Spark?","datePublished":"2023-07-31T11:00:00+00:00","dateModified":"2023-07-18T11:49:54+00:00","mainEntityOfPage":{"@id":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/"},"wordCount":158,"publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"articleSection":["Spark"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/","url":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/","name":"What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism in Spark? - Big Data In Real World","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/#website"},"datePublished":"2023-07-31T11:00:00+00:00","dateModified":"2023-07-18T11:49:54+00:00","description":"Both spark.sql.shuffle.partitions and spark.default.parallelism have an impact on performance and this post explains the difference between the two.","breadcrumb":{"@id":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-spark-sql-shuffle-partitions-and-spark-default-parallelism-in-spark\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.bigdatainrealworld.com\/"},{"@type":"ListItem","position":2,"name":"What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism in Spark?"}]},{"@type":"WebSite","@id":"https:\/\/www.bigdatainrealworld.com\/#website","url":"https:\/\/www.bigdatainrealworld.com\/","name":"Big Data In Real World","description":"Learn Big Data from experts!","publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.bigdatainrealworld.com\/#organization","name":"Big Data In Real World","url":"https:\/\/www.bigdatainrealworld.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","contentUrl":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","width":500,"height":500,"caption":"Big Data In Real World"},"image":{"@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/bigdatainrealworld"]},{"@type":"Person","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67","name":"Big Data In Real World","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","caption":"Big Data In Real World"},"description":"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.","sameAs":["https:\/\/www.bigdatainrealworld.com\/"],"url":"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/"}]}},"_links":{"self":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/2307"}],"collection":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/comments?post=2307"}],"version-history":[{"count":2,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/2307\/revisions"}],"predecessor-version":[{"id":2311,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/2307\/revisions\/2311"}],"wp:attachment":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/media?parent=2307"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/categories?post=2307"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/tags?post=2307"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}