How to list topics without accessing Zookeeper in Kafka?
May 29, 2023How to create and use UDF in Spark?
June 12, 2023If you work with Apache Hadoop, you may find yourself needing to kill multiple YARN applications at once. While you can kill them one by one using the yarn application -kill command, this can be a tedious and time-consuming process. Fortunately, there is a faster way to kill multiple YARN applications at once using the yarn application command in combination with awk.
Here’s a step-by-step guide to killing multiple YARN applications at once using the yarn application command and awk.
List all running YARN applications
First, you need to list all running YARN applications using the yarn application -list command
yarn application -list
This will give you a list of all running YARN applications, including their application IDs, names, user names, and other information.
Method 1: Using the grep and awk Commands to filter applications to kill
Kill multiple YARN applications at once is to use the grep and awk commands to filter and extract the application IDs from the list of running applications. Once you have the application IDs, you can use the yarn application -kill command to stop the applications.
Here’s an example command to kill multiple YARN applications using the grep and awk commands:
yarn application -list | grep <filter-text> | awk '{print $1}' | xargs yarn application -kill
Replace <filter-text> with the text you want to use to filter the running applications. For example, if you want to kill all applications submitted by a specific user, you can use that user’s username as the filter text.
This command will list all the running YARN applications that match the filter text and extract their application IDs using the awk command. The xargs command is used to pass the list of application IDs to the yarn application -kill command, which will stop all the applications.
Note: Be careful when using this command, as it will kill all the applications that match the filter text without prompting for confirmation.
Method 2: Write a script to loop and kill
Below script uses a for loop to get all the applications which are either ACCEPTED or RUNNING and passing the application ids to the YARN application kill command.
Kill all applications on YARN which are in ACCEPTED state:
for x in $(yarn application -list -appStates ACCEPTED | awk 'NR > 2 { print $1 }'); do yarn application -kill $x; done
Kill all applications on YARN which are in RUNNING state:
for x in $(yarn application -list -appStates RUNNING | awk 'NR > 2 { print $1 }'); do yarn application -kill $x; done
Note: Be careful when using both the above commands, as it will kill all the applications that match the filter text without prompting for confirmation.
Conclusion
In conclusion, the yarn application command in combination with awk is a powerful tool for killing multiple YARN applications at once. By following the steps outlined in this guide, you can quickly and easily extract the application IDs of the YARN applications you want to kill and pass them as arguments to the yarn application -kill command. This can save you a lot of time and effort compared to manually killing each application one by one.
However, it’s important to exercise caution when using the yarn application -kill command, as killing running applications can have unintended consequences such as data loss or other system issues. Make sure to only kill applications that you are certain you no longer need, and communicate with other users of the system to avoid conflicts. With these precautions in mind, you can use the yarn application command and awk to efficiently manage your YARN applications.
1 Comment
[…] The Big Data in Real World team doesn’t have time to mess around: […]