Both spark.sql.shuffle.partitions and spark.default.parallelism control the number of tasks that get executed at runtime there by controlling the distribution and parallelism. Which means both properties have a direct effect on performance.<\/p>\n\n\n\n

spark.sql.shuffle.partitions<\/h2>\n\n\n\n

spark.sql.shuffle.partitions configures the number of partitions that are used when shuffling data for joins or aggregations. The default for this this property is 200.<\/p>\n\n\n\n

spark.default.parallelism<\/h2>\n\n\n\n

spark.default.parallelism is the default number of partitions in RDDs returned by transformations like join, reduceByKey, and parallelize when not set explicitly by the user. <\/p>\n\n\n\n

spark.default.parallelism works only with RDDs and is ignored when working with DataFrames.<\/p>\n\n\n\n

Default value depend on your deployment –<\/p>\n\n\n\n

\n
• Local mode: number of cores on the local machine<\/li>\n\n\n\n
• Mesos fine grained mode: 8<\/li>\n\n\n\n
• Others: total number of cores on all executor nodes or 2, whichever is larger<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"

  Both spark.sql.shuffle.partitions and spark.default.parallelism control the number of tasks that get executed at runtime there by controlling the distribution and parallelism. Which means both properties have [\u2026]<\/span><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[],"class_list":["post-2307","post","type-post","status-publish","format-standard","hentry","category-spark"],"yoast_head":"\n