How to kill a running Spark application?

Apache Spark is a powerful open-source distributed computing system used for big data processing. However, sometimes you may need to kill a running Spark application for various reasons, such as if the application is stuck, consuming too many resources, or taking too long to complete. In this post, we will discuss how to kill a running Spark application.

Finding the application ID

To kill a running Spark application, you first need to find its application ID. You can find the application ID by running the following command in the Spark shell:

sc.applicationId

This command will return the application ID, which you should make a note of as you will need it in the next step.

You can also run a yarn command to list all applications and filter the application list using grep like below.

yarn application -list | grep <user-name> | awk ‘{print $1}’

Replace <user-name> with your username. This command will return the application ID, which you should make a note of as you will need it in the next step.

Killing the application

Once you have the application ID, you can use the following command to kill the running Spark application

yarn application -kill <application-id>

Replace <application-id> with the actual application ID you obtained in the previous step. This command will send a kill signal to the running Spark application, causing it to stop.

Verifying the application status

To verify that the Spark application has been killed, you can use the following command:

yarn application -status <application-id>

This command will show the status of the Spark application. If the application has been successfully killed, the status will show as “FAILED” or “KILLED”.

Conclusion

In this post, we discussed how to kill a running Spark application using the yarn application -kill command. Remember to first obtain the application ID using the sc.applicationId command before attempting to kill the application. With this knowledge, you can now easily stop a running Spark application that is causing issues. I hope you find this blog post helpful.