How to specify conditional expressions (OR, AND, NOT) when searching documents in Elasticsearch?
April 14, 2021What is the difference between reduceByKey and aggregateByKey in Spark?
April 19, 2021This post will list out the steps to properly remove a node from a Hadoop cluster. It is not advisable to just shut down the node abruptly.
Node exclusions should be properly recorded in a file that is referred to by the property dfs.hosts.exclude. This property doesn’t have default value so in the absence of a file location and a file, the Hadoop cluster will not exclude any nodes.
When dfs.hosts.exclude is not set
Follow the below steps when dfs.hosts.exclude is not set in your cluster
- Shutdown the Namenode
- Edit hdfs-site.xml add an entry for dfs.hosts.exclude with the location of the file
- Add the hostname to the file mentioned in dfs.hosts.exclude that you are planning to remove
- Start namenode
When dfs.hosts.exclude is already set
Add the hostname to the file mentioned in dfs.hosts.exclude that you are planning to remove
After adding the hostname to the exclusion run the below command to exclude the node from functioning as a Datanode
hdfs dfsadmin -refreshNodes
Below command will exclude the node from functioning as a Node Manager
yarn rmadmin -refreshNodes
Why not just shut down the nodes?
Abruptly shutting down a node will cause the HDFS blocks stored in the nodes to be under replicated and upon shut down HDFS will start replicating the blocks from the available nodes to a new set of nodes to bring the replication to 3 (by default).
Abruptly shutting down a node will also cause the MapReduce and other jobs executing in the cluster to fail abruptly.
Removing the node by excluding as described above would first replicate the blocks in the node that is being removed to other nodes and stop talking new jobs and wait for the running jobs to complete execution. Hence this approach should be followed to safely remove the nodes from the cluster.
