How to kill a running Spark application? - Big Data In Real World

How to kill a running Spark application?

How does a consumer know the offset to read after restart in Kafka?
October 2, 2023
Sunset: Hadoop Developer In Real World cluster
June 2, 2024
How does a consumer know the offset to read after restart in Kafka?
October 2, 2023
Sunset: Hadoop Developer In Real World cluster
June 2, 2024

Apache Spark is a powerful open-source distributed computing system used for big data processing. However, sometimes you may need to kill a running Spark application for various reasons, such as if the application is stuck, consuming too many resources, or taking too long to complete. In this post, we will discuss how to kill a running Spark application.

Finding the application ID

To kill a running Spark application, you first need to find its application ID. You can find the application ID by running the following command in the Spark shell:

sc.applicationId

This command will return the application ID, which you should make a note of as you will need it in the next step.

You can also run a yarn command to list all applications and filter the application list using grep like below.

yarn application -list | grep <user-name> | awk ‘{print $1}’

Replace <user-name> with your username. This command will return the application ID, which you should make a note of as you will need it in the next step.

Killing the application

Once you have the application ID, you can use the following command to kill the running Spark application

yarn application -kill <application-id>

Replace <application-id> with the actual application ID you obtained in the previous step. This command will send a kill signal to the running Spark application, causing it to stop.

Verifying the application status

To verify that the Spark application has been killed, you can use the following command:

yarn application -status <application-id>

This command will show the status of the Spark application. If the application has been successfully killed, the status will show as “FAILED” or “KILLED”.

Conclusion

In this post, we discussed how to kill a running Spark application using the yarn application -kill command. Remember to first obtain the application ID using the sc.applicationId command before attempting to kill the application. With this knowledge, you can now easily stop a running Spark application that is causing issues. I hope you find this blog post helpful. 

Big Data In Real World
Big Data In Real World
We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.

Comments are closed.

How to kill a running Spark application?
This website uses cookies to improve your experience. By using this website you agree to our Data Protection Policy.

Hadoop In Real World is now Big Data In Real World!

X