{"id":1772,"date":"2021-09-15T06:00:00","date_gmt":"2021-09-15T11:00:00","guid":{"rendered":"https:\/\/www.bigdatainrealworld.com\/?p=1772"},"modified":"2023-02-19T07:31:05","modified_gmt":"2023-02-19T13:31:05","slug":"what-are-accumulators-in-spark-when-and-when-not-to-use-them","status":"publish","type":"post","link":"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/","title":{"rendered":"What are accumulators in Spark, when and when not to use them?"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">Accumulators are like global variables in Spark application. In the real world, accumulators are used as counters and keep to keep track of something at an application level. Accumulators serve a very similar purpose as counters in MapReduce.<\/span><br \/>\n<script src=\"https:\/\/embed.lpcontent.net\/leadboxes\/current\/embed.js\" async=\"\" defer=\"defer\"><\/script><\/p>\n<p style=\"border: 1px; border-style: solid; border-color: #005cb9; background-color: #f1f5f9; padding: 1em;\">Do you like us to send you a 47 page Definitive guide on Spark join algorithms? ===&gt; <button style=\"background: #afbf00; border-color: #afbf00; border-radius: 20px; color: #ffffff; display: inline-block; vertical-align: middle; padding: 16px 32px; min-width: 192px; border: 1px solid #afbf00; font-size: 1rem; font-family: Helvetica, Arial, sans-serif; text-align: center; outline: 0; line-height: 1; cursor: pointer; -webkit-transition: background 0.3s, color 0.3s, border 0.3s; transition: background 0.3s, color 0.3s, border 0.3s; box-shadow: 0px 2px 5px rgba(0, 0, 0, 0.6);\" data-leadbox-popup=\"iRwJt3SrgaavPhqQ3dnrtg\" data-leadbox-domain=\"ccubecompany.lpages.co\">Send me the guide<\/button><\/p>\n<h2><span style=\"font-weight: 400;\">Example<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Let\u2019s say we have a sales dataset and we want to count the number of \u201cbad\u201d records. We consider a record to be bad when the sale_quantity column is 0. Because a sale record doesn\u2019t make sense with 0 quantity.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">First we register a Long accumulator variable named badRecords.<\/span><\/p>\n<pre class=\"lang:default decode:true \">scala&gt; val badRecords = spark.sparkContext.longAccumulator(\"badRecords\")\nbadRecords: org.apache.spark.util.LongAccumulator = LongAccumulator(id: 28, name: Some(badRecords), value: 0)<\/pre>\n<p><span style=\"font-weight: 400;\">We can call the value() on the accumulator and we get the value as 0<\/span><\/p>\n<pre class=\"lang:default decode:true \">scala&gt; badRecords.value\nres2: Long = 0\n\n<\/pre>\n<p><span style=\"font-weight: 400;\">Let\u2019s now create a quick list of sale record and create a dataframe out of the data. You can see that record number 2 and 5 have 0 quantity and we consider them as \u201cbad\u201d records.<\/span><\/p>\n<pre class=\"lang:default decode:true \">scala&gt; &nbsp; val data = Seq(\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| &nbsp; &nbsp; (1, \"Shirt\", 20, 2),\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| &nbsp; &nbsp; (2, \"Pants\", 30, 0),\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| &nbsp; &nbsp; (3, \"Skirt\", 40, 3),\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| &nbsp; &nbsp; (4, \"Hat\", 10, 4),\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| &nbsp; &nbsp; (5, \"Shirt\", 20, 0)\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| &nbsp; )\ndata: Seq[(Int, String, Int, Int)] = List((1,Shirt,20,2), (2,Pants,30,0), (3,Skirt,40,3), (4,Hat,10,4), (5,Shirt,20,0))\n\nscala&gt; val columns = Seq(\"sale_id\",\"sale_item\",\"sale_price\", \"sale_quantity\")\ncolumns: Seq[String] = List(sale_id, sale_item, sale_price, sale_quantity)\n\nscala&gt; val df = data.toDF(columns:_*)\ndf: org.apache.spark.sql.DataFrame = [sale_id: int, sale_item: string ... 2 more fields]<\/pre>\n<p>Here we are calling the foreach() on our dataframe and when we encounter 0 quantity in the sales records. We call the add() to increment the badRecords accumulator count by 1. Now when we print the value of the accumulator we get 2. Which is correct.<\/p>\n<pre class=\"lang:default decode:true \">scala&gt; &nbsp; df.foreach(row =&gt; {\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| &nbsp; &nbsp; &nbsp; if(row.getInt(3) == 0) badRecords.add(1)\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| &nbsp; &nbsp; }\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| &nbsp; )\n\nscala&gt; badRecords.value\nres4: Long = 2<\/pre>\n<p>&nbsp;<\/p>\n<h2><span style=\"font-weight: 400;\">When to use accumulators?<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Now we know the function of accumulators, let&#8217;s look at the correct way to use accumulators. If you notice, we have used accumulators inside the foreach(). foreach() is an action and action functions are the right location to use accumulators.&nbsp;<\/span><\/p>\n<p><span style=\"font-weight: 400;\">We have a separate post on foreach() and click here to learn more about foreach().<\/span><\/p>\n<p><span style=\"font-weight: 400;\">foreach() are when to update the state of external variables and systems aka. it causes side effects and foreach() is an appropriate place to update accumulators and more over foreach() is an action function and not a transformation function and hence it is the correct place to manipulate accumulators.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">When NOT to use accumulators?<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Accumulators should not be used inside map() functions doing so can have unintended consequences.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Spark can rerun a task in a few instances &#8211;<\/span><\/p>\n<ol>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">When a task encounters an exception, Spark will re-execute the task 4 times by default.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">If an executor crashes, Spark will re execute the tasks&nbsp;<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">If a task is running slow, Spark can rerun another copy of the task and this is called speculative execution. It only takes results from the task which completes first.<\/span><\/li>\n<\/ol>\n<p><span style=\"font-weight: 400;\">When a task re-execute, it will execute all the transformation functions in the task and this causes the accumulator value which was already manipulated by the first execution of the task to get manipulated again causing duplication in the accumulator\u2019s result.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Due to this reason, always include code related to accumulator in action functions like foreach(). Spark will not complain at compile time or runtime when you include code related to accumulator in transformation functions like map() so make sure to keep this point in mind when you deal with accumulators.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Accumulators are like global variables in Spark application. In the real world, accumulators are used as counters and keep to keep track of something at an<span class=\"excerpt-hellip\"> [\u2026]<\/span><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[],"class_list":["post-1772","post","type-post","status-publish","format-standard","hentry","category-spark"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>What are accumulators in Spark, when and when not to use them? - Big Data In Real World<\/title>\n<meta name=\"description\" content=\"In this post learn all there is to know about accumulators in Spark and more importantly learn the right way to use them.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What are accumulators in Spark, when and when not to use them? - Big Data In Real World\" \/>\n<meta property=\"og:description\" content=\"In this post learn all there is to know about accumulators in Spark and more importantly learn the right way to use them.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/\" \/>\n<meta property=\"og:site_name\" content=\"Big Data In Real World\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/bigdatainrealworld\" \/>\n<meta property=\"article:published_time\" content=\"2021-09-15T11:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-02-19T13:31:05+00:00\" \/>\n<meta name=\"author\" content=\"Big Data In Real World\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Big Data In Real World\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/\"},\"author\":{\"name\":\"Big Data In Real World\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\"},\"headline\":\"What are accumulators in Spark, when and when not to use them?\",\"datePublished\":\"2021-09-15T11:00:00+00:00\",\"dateModified\":\"2023-02-19T13:31:05+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/\"},\"wordCount\":498,\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"articleSection\":[\"Spark\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/\",\"name\":\"What are accumulators in Spark, when and when not to use them? - Big Data In Real World\",\"isPartOf\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\"},\"datePublished\":\"2021-09-15T11:00:00+00:00\",\"dateModified\":\"2023-02-19T13:31:05+00:00\",\"description\":\"In this post learn all there is to know about accumulators in Spark and more importantly learn the right way to use them.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.bigdatainrealworld.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What are accumulators in Spark, when and when not to use them?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#website\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"name\":\"Big Data In Real World\",\"description\":\"Learn Big Data from experts!\",\"publisher\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#organization\",\"name\":\"Big Data In Real World\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"contentUrl\":\"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png\",\"width\":500,\"height\":500,\"caption\":\"Big Data In Real World\"},\"image\":{\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/bigdatainrealworld\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67\",\"name\":\"Big Data In Real World\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g\",\"caption\":\"Big Data In Real World\"},\"description\":\"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.\",\"sameAs\":[\"https:\/\/www.bigdatainrealworld.com\/\"],\"url\":\"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What are accumulators in Spark, when and when not to use them? - Big Data In Real World","description":"In this post learn all there is to know about accumulators in Spark and more importantly learn the right way to use them.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/","og_locale":"en_US","og_type":"article","og_title":"What are accumulators in Spark, when and when not to use them? - Big Data In Real World","og_description":"In this post learn all there is to know about accumulators in Spark and more importantly learn the right way to use them.","og_url":"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/","og_site_name":"Big Data In Real World","article_publisher":"https:\/\/www.facebook.com\/bigdatainrealworld","article_published_time":"2021-09-15T11:00:00+00:00","article_modified_time":"2023-02-19T13:31:05+00:00","author":"Big Data In Real World","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Big Data In Real World","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/#article","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/"},"author":{"name":"Big Data In Real World","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67"},"headline":"What are accumulators in Spark, when and when not to use them?","datePublished":"2021-09-15T11:00:00+00:00","dateModified":"2023-02-19T13:31:05+00:00","mainEntityOfPage":{"@id":"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/"},"wordCount":498,"publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"articleSection":["Spark"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/","url":"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/","name":"What are accumulators in Spark, when and when not to use them? - Big Data In Real World","isPartOf":{"@id":"https:\/\/www.bigdatainrealworld.com\/#website"},"datePublished":"2021-09-15T11:00:00+00:00","dateModified":"2023-02-19T13:31:05+00:00","description":"In this post learn all there is to know about accumulators in Spark and more importantly learn the right way to use them.","breadcrumb":{"@id":"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.bigdatainrealworld.com\/what-are-accumulators-in-spark-when-and-when-not-to-use-them\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.bigdatainrealworld.com\/"},{"@type":"ListItem","position":2,"name":"What are accumulators in Spark, when and when not to use them?"}]},{"@type":"WebSite","@id":"https:\/\/www.bigdatainrealworld.com\/#website","url":"https:\/\/www.bigdatainrealworld.com\/","name":"Big Data In Real World","description":"Learn Big Data from experts!","publisher":{"@id":"https:\/\/www.bigdatainrealworld.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.bigdatainrealworld.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.bigdatainrealworld.com\/#organization","name":"Big Data In Real World","url":"https:\/\/www.bigdatainrealworld.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","contentUrl":"https:\/\/www.bigdatainrealworld.com\/wp-content\/uploads\/2023\/02\/black.png","width":500,"height":500,"caption":"Big Data In Real World"},"image":{"@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/bigdatainrealworld"]},{"@type":"Person","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/24cab2292ef49c73053440c86515ef67","name":"Big Data In Real World","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.bigdatainrealworld.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d332bc24fe9b3182f0a22135f163ac4e?s=96&d=retro&r=g","caption":"Big Data In Real World"},"description":"We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.","sameAs":["https:\/\/www.bigdatainrealworld.com\/"],"url":"https:\/\/www.bigdatainrealworld.com\/author\/bigdatainrealworld\/"}]}},"_links":{"self":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1772"}],"collection":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/comments?post=1772"}],"version-history":[{"count":3,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1772\/revisions"}],"predecessor-version":[{"id":2088,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/posts\/1772\/revisions\/2088"}],"wp:attachment":[{"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/media?parent=1772"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/categories?post=1772"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bigdatainrealworld.com\/wp-json\/wp\/v2\/tags?post=1772"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}