How to kill multiple YARN applications at once? - Big Data In Real World

How to kill multiple YARN applications at once?

How to list topics without accessing Zookeeper in Kafka?
May 29, 2023
How to create and use UDF in Spark?
June 12, 2023
How to list topics without accessing Zookeeper in Kafka?
May 29, 2023
How to create and use UDF in Spark?
June 12, 2023

If you work with Apache Hadoop, you may find yourself needing to kill multiple YARN applications at once. While you can kill them one by one using the yarn application -kill command, this can be a tedious and time-consuming process. Fortunately, there is a faster way to kill multiple YARN applications at once using the yarn application command in combination with awk.

Here’s a step-by-step guide to killing multiple YARN applications at once using the yarn application command and awk.

List all running YARN applications

First, you need to list all running YARN applications using the yarn application -list command

yarn application -list

This will give you a list of all running YARN applications, including their application IDs, names, user names, and other information.

Method 1: Using the grep and awk Commands to filter applications to kill

Kill multiple YARN applications at once is to use the grep and awk commands to filter and extract the application IDs from the list of running applications. Once you have the application IDs, you can use the yarn application -kill command to stop the applications.

Here’s an example command to kill multiple YARN applications using the grep and awk commands:

yarn application -list | grep <filter-text> | awk '{print $1}' | xargs yarn application -kill

Replace <filter-text> with the text you want to use to filter the running applications. For example, if you want to kill all applications submitted by a specific user, you can use that user’s username as the filter text.

This command will list all the running YARN applications that match the filter text and extract their application IDs using the awk command. The xargs command is used to pass the list of application IDs to the yarn application -kill command, which will stop all the applications.

Note: Be careful when using this command, as it will kill all the applications that match the filter text without prompting for confirmation.

Method 2: Write a script to loop and kill

Below script uses a for loop to get all the applications which are either ACCEPTED or RUNNING and passing the application ids to the YARN application kill command.

Kill all applications on YARN which are in ACCEPTED state:

for x in $(yarn application -list -appStates ACCEPTED | awk 'NR > 2 { print $1 }'); do yarn application -kill $x; done

Kill all applications on YARN which are in RUNNING state:

for x in $(yarn application -list -appStates RUNNING | awk 'NR > 2 { print $1 }'); do yarn application -kill $x; done

Note: Be careful when using both the above commands, as it will kill all the applications that match the filter text without prompting for confirmation.

Conclusion

In conclusion, the yarn application command in combination with awk is a powerful tool for killing multiple YARN applications at once. By following the steps outlined in this guide, you can quickly and easily extract the application IDs of the YARN applications you want to kill and pass them as arguments to the yarn application -kill command. This can save you a lot of time and effort compared to manually killing each application one by one.

However, it’s important to exercise caution when using the yarn application -kill command, as killing running applications can have unintended consequences such as data loss or other system issues. Make sure to only kill applications that you are certain you no longer need, and communicate with other users of the system to avoid conflicts. With these precautions in mind, you can use the yarn application command and awk to efficiently manage your YARN applications.

Big Data In Real World
Big Data In Real World
We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.

1 Comment

  1. […] The Big Data in Real World team doesn’t have time to mess around: […]

How to kill multiple YARN applications at once?
This website uses cookies to improve your experience. By using this website you agree to our Data Protection Policy.

Hadoop In Real World is now Big Data In Real World!

X