How to properly remove or decommission a node from an Elasticsearch cluster?

Shutting down a node abruptly is not the right way to decommission or remove a node from the Elasticsearch cluster. Doing so will cause your shards which are replicated to go down in replication and it could cause disruption to the clients who are currently consuming data from Elasticsearch.

Proper way to decommission or remove a node from Elasticsearch is to add the host to the exclusion list.

Adding a node to the exclusion list

Issue the below PUT on _cluster/settings with the cluster.routing.allocation.exclude._ip property along with the IP address of the host you want to exclude.

This will cause Elasticsearch to allocate the shards on that node to the remaining nodes at the same time, not abruptly crashing the clients consuming data from the node.

Once all the shards have been reallocated we can shutdown the node.

curl -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
  "transient" : {
    "cluster.routing.allocation.exclude._ip" : "178.2.3.1"
  }
}'

If we have multiple IPs to exclude we can use a wildcard in the exclusion IPs.