{"id":1902,"date":"2021-04-07T06:00:00","date_gmt":"2021-04-07T11:00:00","guid":{"rendered":"https:\/\/www.bigdatainrealworld.com\/?p=1902"},"modified":"2023-02-19T07:31:31","modified_gmt":"2023-02-19T13:31:31","slug":"what-is-the-difference-between-groupbykey-and-reducebykey-in-spark","status":"publish","type":"post","link":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/","title":{"rendered":"What is the difference between groupByKey and reduceByKey in Spark?"},"content":{"rendered":"\n<p>Both reduceByKey and groupByKey result in wide transformations which means both triggers a shuffle operation.<\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">The key difference between reduceByKey and groupByKey is that reduceByKey does a map side combine and groupByKey does not do a map side combine.<\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">Let\u2019s say we are computing word count on a file with below line<\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">RED GREEN RED RED<\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">At runtime let\u2019s say we end up with 2 partitions.<\/span><\/p>\n\n\n\n<pre class=\"wp-block-preformatted lang:default decode:true\">Partition 1\nRED\nGREEN\nPartition 2\nRED\nRED<\/pre>\n\n\n\n<p><script src=\"https:\/\/embed.lpcontent.net\/leadboxes\/current\/embed.js\" async=\"\" defer=\"defer\"><\/script><\/p>\n\n\n\n<p>Do you like us to send you a 47 page Definitive guide on Spark join algorithms? ===&gt; <button style=\"background: #afbf00; border-color: #afbf00; border-radius: 20px; color: #ffffff; display: inline-block; vertical-align: middle; padding: 16px 32px; min-width: 192px; border: 1px solid #afbf00; font-size: 1rem; font-family: Helvetica, Arial, sans-serif; text-align: center; outline: 0; line-height: 1; cursor: pointer; -webkit-transition: background 0.3s, color 0.3s, border 0.3s; transition: background 0.3s, color 0.3s, border 0.3s; box-shadow: 0px 2px 5px rgba(0, 0, 0, 0.6);\" data-leadbox-popup=\"iRwJt3SrgaavPhqQ3dnrtg\" data-leadbox-domain=\"ccubecompany.lpages.co\">Send me the guide<\/button><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span style=\"font-weight: 400;\">groupByKey()<\/span><\/h2>\n\n\n\n<p><span style=\"font-weight: 400;\">With groupByKey() on the above word count problem, since we have two partitions we will end up with 2 tasks. The output of the tasks will look like below &#8211;<\/span><\/p>\n\n\n\n<pre class=\"wp-block-preformatted lang:default decode:true\">Task 1\nRED, 1\nGREEN, 1\nTask 2\nRED, 1\nRED, 1<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">All the 4 elements from Task 1 and 2 will be sent over the network to the Task performing the reduce operation.<\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">Task performing reduce<\/span><\/p>\n\n\n\n<pre class=\"wp-block-preformatted lang:default decode:true\">RED, 1\nGREEN, 1\nRED, 1\nRED, 1<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">Yielding the result<\/span><\/p>\n\n\n\n<pre class=\"wp-block-preformatted lang:default decode:true\">GREEN 1\nRED 3<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">There are two problems with this &#8211;<\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">Since the data is not combined or reduced on the map side, we transferred all elements over the network during shuffle.<\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">Since all elements are sent to the task performing the aggregate operation, the number of elements to be handled by the task will be more and could possibly result in an Out of Memory exception.<\/span><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span style=\"font-weight: 400;\">reduceByKey()<\/span><\/h2>\n\n\n\n<p><span style=\"font-weight: 400;\">reduceByKey is optimized with a map side combine.<\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">Just like groupByKey(), on the same word count problem, since we have two partitions we will end up with 2 tasks. However with a map side combine, the output of the tasks will look like below &#8211;<\/span><\/p>\n\n\n\n<pre class=\"wp-block-preformatted lang:default decode:true\">Task 1\nRED, 1\nGREEN, 1\nTask 2\nRED, 2<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">With reduceByKey only 3 elements from Task 1 and 2 will be sent over the network to the Task performing the reduce operation.<\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">Task performing reduce<\/span><\/p>\n\n\n\n<pre class=\"wp-block-preformatted lang:default decode:true\">RED, 1\nGREEN, 1\nRED, 2<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">Yielding the result<\/span><\/p>\n\n\n\n<pre class=\"wp-block-preformatted lang:default decode:true\">GREEN 1\nRED 3<\/pre>\n\n\n\n<p><span style=\"font-weight: 400;\">reduceByKey() is optimized due to the map side combine. There by sending fewer elements over the network.<\/span><\/p>\n\n\n\n<p><span style=\"font-weight: 400;\">Also fewer elements are reduced on the task performing the reduce operation after the shuffle.<\/span><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span style=\"font-weight: 400;\">Should I use groupByKey or reduceByKey?<\/span><\/h2>\n\n\n\n<p><span style=\"font-weight: 400;\">Use reduceByKey as it performs map side combine which reduces the amount of data sent over the network during shuffle and thereby also reduces the amount of data reduced.<\/span><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span style=\"font-weight: 400;\">Is there a good reason to use groupByKey?<\/span><\/h2>\n\n\n\n<p><span style=\"font-weight: 400;\">Use groupByKey() when map side combine is not necessary and could cause adverse effects to the output.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Both reduceByKey and groupByKey result in wide transformations which means both triggers a shuffle operation. The key difference between reduceByKey and groupByKey is that reduceByKey does<span class=\"excerpt-hellip\"> [\u2026]<\/span><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[],"class_list":["post-1902","post","type-post","status-publish","format-standard","hentry","category-spark"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>What is the difference between groupByKey and reduceByKey in Spark? - Big Data In Real World<\/title>\n<meta name=\"description\" content=\"A quick 5 minute read to understand the difference between groupByKey and reduceByKey in Spark.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What is the difference between groupByKey and reduceByKey in Spark? - Big Data In Real World\" \/>\n<meta property=\"og:description\" content=\"A quick 5 minute read to understand the difference between groupByKey and reduceByKey in Spark.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/\" \/>\n<meta property=\"og:site_name\" content=\"Big Data In Real World\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/bigdatainrealworld\" \/>\n<meta property=\"article:published_time\" content=\"2021-04-07T11:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-02-19T13:31:31+00:00\" \/>\n<meta name=\"author\" content=\"Big Data In Real World\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Big Data In Real World\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/\"},\"author\":{\"name\":\"Big Data In Real World\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\"},\"headline\":\"What is the difference between groupByKey and reduceByKey in Spark?\",\"datePublished\":\"2021-04-07T11:00:00+00:00\",\"dateModified\":\"2023-02-19T13:31:31+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/\"},\"wordCount\":378,\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"articleSection\":[\"Spark\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/\",\"name\":\"What is the difference between groupByKey and reduceByKey in Spark? - Big Data In Real World\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\"},\"datePublished\":\"2021-04-07T11:00:00+00:00\",\"dateModified\":\"2023-02-19T13:31:31+00:00\",\"description\":\"A quick 5 minute read to understand the difference between groupByKey and reduceByKey in Spark.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.bigdatainrealworld.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What is the difference between groupByKey and reduceByKey in Spark?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"name\":\"Big Data In Real World\",\"description\":\"Learn Big Data from experts!\",\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\",\"name\":\"Big Data In Real World\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"contentUrl\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"width\":500,\"height\":500,\"caption\":\"Big Data In Real World\"},\"image\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/bigdatainrealworld\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\",\"name\":\"Big Data In Real World\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"caption\":\"Big Data In Real World\"},\"description\":\"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.\",\"sameAs\":[\"https:\/\/www.bigdatainrealworld.com\/\"],\"url\":\"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What is the difference between groupByKey and reduceByKey in Spark? - Big Data In Real World","description":"A quick 5 minute read to understand the difference between groupByKey and reduceByKey in Spark.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/","og_locale":"en_US","og_type":"article","og_title":"What is the difference between groupByKey and reduceByKey in Spark? - Big Data In Real World","og_description":"A quick 5 minute read to understand the difference between groupByKey and reduceByKey in Spark.","og_url":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/","og_site_name":"Big Data In Real World","article_publisher":"https:\/\/www.facebook.com\/bigdatainrealworld","article_published_time":"2021-04-07T11:00:00+00:00","article_modified_time":"2023-02-19T13:31:31+00:00","author":"Big Data In Real World","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Big Data In Real World","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/#article","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/"},"author":{"name":"Big Data In Real World","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67"},"headline":"What is the difference between groupByKey and reduceByKey in Spark?","datePublished":"2021-04-07T11:00:00+00:00","dateModified":"2023-02-19T13:31:31+00:00","mainEntityOfPage":{"@id":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/"},"wordCount":378,"publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"articleSection":["Spark"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/","url":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/","name":"What is the difference between groupByKey and reduceByKey in Spark? - Big Data In Real World","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/#website"},"datePublished":"2021-04-07T11:00:00+00:00","dateModified":"2023-02-19T13:31:31+00:00","description":"A quick 5 minute read to understand the difference between groupByKey and reduceByKey in Spark.","breadcrumb":{"@id":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.bigdatainrealworld.com\/what-is-the-difference-between-groupbykey-and-reducebykey-in-spark\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.bigdatainrealworld.com\/"},{"@type":"ListItem","position":2,"name":"What is the difference between groupByKey and reduceByKey in Spark?"}]},{"@type":"WebSite","@id":"https:\/\/www.bigdatainrealworld.com\/#website","url":"https:\/\/www.bigdatainrealworld.com\/","name":"Big Data In Real World","description":"Learn Big Data from experts!","publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.bigdatainrealworld.com\/#organization","name":"Big Data In Real World","url":"https:\/\/www.bigdatainrealworld.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","contentUrl":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","width":500,"height":500,"caption":"Big Data In Real World"},"image":{"@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/bigdatainrealworld"]},{"@type":"Person","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67","name":"Big Data In Real World","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","caption":"Big Data In Real World"},"description":"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.","sameAs":["https:\/\/www.bigdatainrealworld.com\/"],"url":"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/"}]}},"_links":{"self":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1902"}],"collection":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/comments?post=1902"}],"version-history":[{"count":4,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1902\/revisions"}],"predecessor-version":[{"id":2043,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1902\/revisions\/2043"}],"wp:attachment":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/media?parent=1902"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/categories?post=1902"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/tags?post=1902"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}