Spark

Over time we have written several posts on Spark joins and join algorithms explaining the internal working of these join algorithms. Here are all the posts […]

Shuffle-and-Replication does not mean a “true” shuffle as in records with the same keys are sent to the same partition. Instead the entire partition of the […]

Broadcast Nested Loop join works by broadcasting one of the entire datasets and performing a nested loop to join the data. So essentially every record from […]

Take a look at the below execution plan. Currently when you print the executed plan, you see that Spark is using Sort Merge Join. scala> dfJoined.queryExecution.executedPlan […]

There are several factors Spark takes into account before deciding on the type of join algorithm to use to join datasets at runtime. Spark has the […]

Let’s create RDD first. Below we are creating a RDD with some sample data. scala>   val data = Seq(      |     (1, "Shirt", 20, […]

With Spark 3.0, we can specify the type of join algorithm we would like Spark to use at runtime. Do you like us to send you […]

Cartesian Product Join (a.k.a Shuffle-and-Replication Nested Loop) join works very similar to a Broadcast Nested Loop join except the dataset is not broadcasted.  Shuffle-and-Replication does not […]

Spark spits out a lot of INFO messages in the logs. These messages stand in the way when you are troubleshooting an error and looking for […]

Shuffle Sort Merge Join, as the name indicates, involves a sort operation. Shuffle Sort Merge Join has 3 phases. Shuffle Phase – both datasets are shuffled […]

Broadcast Hash Join in Spark works by broadcasting the small dataset to all the executors and once the data is broadcasted a standard hash join is […]

Broadcast Nested Loop join works by broadcasting one of the entire datasets and performing a nested loop to join the data. So essentially every record from […]

Shuffle Hash Join, as the name indicates works by shuffling both datasets. So the same keys from both sides end up in the same partition or […]

This is a very common use case, if you are working on a Spark project and writing code in Java. You have a list and now […]

The function of repartition and coalesce functions in Spark is to change the number of partitions on a DataFrame. Repartition Increase or decrease partitions. Repartition always […]

Most often when we are trying to work with data in Spark we might want to preview the data or the solution in Spark shell right […]