Spark

Apache Spark is a powerful open-source distributed computing system used for big data processing. However, sometimes you may need to kill a running Spark application for […]

This is going to be a short post.  Number of executors in YARN deployments Spark.executor.instances controls the number of executors in YARN. By default, the number […]

Number of cores spark.executor.cores controls the number of cores available for the executors.  By default, it is 1 core per executor in YARN and all available […]

Apache Spark, the popular distributed computing framework, has been widely adopted for processing large-scale data. With the release of Apache Spark 3.0, a groundbreaking feature called […]

Apache Spark, the popular distributed computing framework, has taken a significant leap forward with the release of Apache Spark 3.0. Packed with new features and enhancements, […]

Both spark.sql.shuffle.partitions and spark.default.parallelism control the number of tasks that get executed at runtime there by controlling the distribution and parallelism. Which means both properties have […]

This post aims at describing the differences between client and cluster deploy modes in Spark. Client mode Cluster mode

In this post we are going to create a Spark UDF which converts temperature from Fahrenheit to Celsius. Here is our data. We have day and […]

Here is our data. We have an employee DataFrame with 3 columns, name, project and cost_to_project. An employee can belong to multiple projects and for each […]

Unfortunately there is no built in function to transpose a DataFrame from columns to rows in Spark. In this post we will show an easy way […]

In this post we will look at the differences between map and mapValues functions and when it is appropriate to use either one. We have a […]

We will be using the spark-xml package from Databrick to read and write XML files with Spark. Here is how we enter the spark shell to […]

In this post we are going to see how to work with Excel files in Spark. We will be using the spark-excel package created by Crealytics. […]

Let’s say we have a DataFrame like below. +---------+-------+---------------+ |  Project|   Name|Cost_To_Project| +---------+-------+---------------+ |Ingestion|  Jerry|           1000| |Ingestion|   Arya|   […]

Let’s say we have a DataFrame with the employee name, project and the cost of the employee to the project. From this data, we have a […]

In this post we are going to describe how to pivot and unpivot a DataFrame in Spark.  We have an employee DataFrame with 3 columns, name, […]

stack function in Spark takes a number of rows as an argument followed by expressions. stack(n, expr1, expr2.. exprn) stack function will generate n rows by […]

Both map and flatMap functions are transformation functions. When applied on RDD, map and flatMap transform each element inside the rdd to something. Consider this simple […]

A quick answer that might come to your mind is to call the count() function on the dataframe and check if the count is greater than […]