Spark

foreach() and foreachPartition() are action function and not transform function. Both functions, since they are actions, they don’t return a RDD back. Do you like us […]

We get a lot of questions on the differences in Spark applications, jobs, stages and tasks. Also we see there is a lot of misunderstanding about […]

It is quite common to see 200 tasks in one of your stages and more specifically at a stage which requires wide transformation. The reason for […]

Let’s see this with an example. Here is our series of instructions in our Spark code. Let’s see how Spark decide on stages and tasks with […]

Accumulators are like global variables in Spark application. In the real world, accumulators are used as counters and keep to keep track of something at an […]

We are sure you have seen the below progress bar before in either Spark shell or while refreshing the Spark UI as your Spark job execution […]

Pretty simple. Use the except() to subtract or find the difference between two dataframes. Do you like us to send you a 47 page Definitive guide […]

java.net.BindException is a common exception when Spark is trying to initialize SparkContext. This is especially a common error when you try to run Spark locally. 16/01/04 […]

RDD, Dataframe and Dataset are all Spark APIs introduced in Spark at different points in time. The goal of these API is to help us work […]

Sometimes you might see a stage being skipped in the DAG visualization in Spark web UI. In this post we are going to discuss couple of […]

In this post we will see how Spark decides the number of tasks and number of tasks to execute in parallel in a job. Let’s see […]

Broadcast variables are variables which are available in all executors executing the Spark application. These variables are already cached and ready to be used by tasks […]

It is a pretty common use case to find the list of duplicate elements or rows in a Spark DataFrame and it is very easy to […]

We get this question a lot so we thought we would write a small post to answer this question. Spark leverages Hadoop’s InputFileFormat to read files […]

Both map and mapPartitions are narrow transformation functions. Both functions don’t trigger a shuffle. Let’s say our RDD has 5 partitions and 10 elements in each […]

reduceByKey() reduceByKey() has the below properties The result of the combination (e.g. a sum) is of the same type that the values The operation when combined […]

Both reduceByKey and groupByKey result in wide transformations which means both triggers a shuffle operation. The key difference between reduceByKey and groupByKey is that reduceByKey does […]

cache() and persist() functions are used to cache intermediate results of a RDD or DataFrame or Dataset. You can mark an RDD, DataFrame or Dataset to […]

Here are three high level reasons using cache() or persist() functions would be appropriate and helpful. Performing multiple actions on the same RDD numbersRDD is used […]