FSNamesystem initialization failed
January 16, 2017Change field termination value in Hive
January 23, 2017DataNode process killed due to Incompatible clusterIDs error
This blog post will describe how to address Incompatible clusterIDs with DataNodes.
2013-04-11 16:26:15,720 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-474150866-127.0.1.1-1365686732002 (storage id DS-317990214-127.0.1.1-50010-1365505141363) service to localhost/127.0.0.1:8020 java.io.IOException: Incompatible clusterIDs in /home/hadoop/workspace/hadoop_space/hadoop23/dfs/data: namenode clusterID = CID-1745a89c-fb08-40f0-a14d-d37d01f199c3; datanode clusterID = CID-bb3547b0-03e4-4588-ac25-f0299ff81e4f at org.apache.hadoop.hdfs.server.datanode.DataStorage .doTransition(DataStorage.java:391) at org.apache.hadoop.hdfs.server.datanode.DataStorage .recoverTransitionRead(DataStorage.java:191) at org.apache.hadoop.hdfs.server.datanode.DataStorage .recoverTransitionRead(DataStorage.java:219) at org.apache.hadoop.hdfs.server.datanode.DataNode.in itStorage(DataNode.java:850) at org.apache.hadoop.hdfs.server.datanode.DataNode.in itBlockPool(DataNode.java:821) at org.apache.hadoop.hdfs.server.datanode.BPOfferServ ice.verifyAndSetNamespaceInfo(BPOfferService.java: 280) at org.apache.hadoop.hdfs.server.datanode.BPServiceAc tor.connectToNNAndHandshake(BPServiceActor.java:22 2) at org.apache.hadoop.hdfs.server.datanode.BPServiceAc tor.run(BPServiceActor.java:664) at java.lang.Thread.run(Thread.java:722) 2013-04-11 16:26:16,212 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool BP-474150866-127.0.1.1-1365686732002 (storage id DS-317990214-127.0.1.1-50010-1365505141363) service to localhost/127.0.0.1:8020 2013-04-11 16:26:16,276 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool BP-474150866-127.0.1.1-1365686732002 (storage id DS-317990214-127.0.1.1-50010-1365505141363) 2013-04-11 16:26:18,396 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode 2013-04-11 16:26:18,940 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0 2013-04-11 16:26:19,668 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: /************************************************** ********** SHUTDOWN_MSG: Shutting down DataNode at user-VirtualBox/127.0.1.1 ************************************************** **********/
Problem
The problem could be that the NameNode was formatted with the below command after the cluster was set up and the data nodes were not formatted, so all the DataNodes are still referring to the old NameNode id causing the error.
sudo -u hdfs hdfs namenode -format
It is a bad idea to format the NameNode in a functioning cluster. It is like wiping out your hard disk. If you are still in learning phase and experimenting then this error makes more sense and not so critical (hopefully !)
Solution
Locate the data directory of the DataNodes. You can find the directory location in hdfs-site.xml , look for property dfs.datanode.data.dir
<property> <name>dfs.datanode.data.dir</name> <value>file:///data/1/dfs/dn</value> </property>
Since the NameNode is formatted you don’t have an option other than formatting (deleting) the contents in DataNode as well. So go to the directory location specified in dfs.datanode.data.dir and issue a delete on that directory across all DataNodes.
Once the data directories are formatted in all DataNodes, restart the DataNode process on all DataNodes and you will be set.