How to change the number of replicas of a Kafka topic?
August 30, 2021How does LATERAL VIEW work in Hive?
September 3, 2021You are getting the below error during DataNode startup. This post talks about how to fix the issue.
2013-04-11 16:25:50,515 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2013-04-11 16:25:50,631 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting 2013-04-11 16:26:15,068 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/hadoop/workspace/hadoop_space/hadoop23/dfs/data/in_use.lock acquired by nodename 3099@user-VirtualBox 2013-04-11 16:26:15,720 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-474150866-127.0.1.1-1365686732002 (storage id DS-317990214-127.0.1.1-50010-1365505141363) service to localhost/127.0.0.1:8020 java.io.IOException: Incompatible clusterIDs in /home/hadoop/workspace/hadoop_space/hadoop23/dfs/data: namenode clusterID = CID-1745a89c-fb08-40f0-a14d-d37d01f199c3; datanode clusterID = CID-bb3547b0-03e4-4588-ac25-f0299ff81e4f at org.apache.hadoop.hdfs.server.datanode.DataStorage .doTransition(DataStorage.java:391) at org.apache.hadoop.hdfs.server.datanode.DataStorage .recoverTransitionRead(DataStorage.java:191) at org.apache.hadoop.hdfs.server.datanode.DataStorage .recoverTransitionRead(DataStorage.java:219) at org.apache.hadoop.hdfs.server.datanode.DataNode.in itStorage(DataNode.java:850) at org.apache.hadoop.hdfs.server.datanode.DataNode.in itBlockPool(DataNode.java:821) at org.apache.hadoop.hdfs.server.datanode.BPOfferServ ice.verifyAndSetNamespaceInfo(BPOfferService.java: 280) at org.apache.hadoop.hdfs.server.datanode.BPServiceAc tor.connectToNNAndHandshake(BPServiceActor.java:22 2) at org.apache.hadoop.hdfs.server.datanode.BPServiceAc tor.run(BPServiceActor.java:664) at java.lang.Thread.run(Thread.java:722) 2013-04-11 16:26:16,212 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool BP-474150866-127.0.1.1-1365686732002 (storage id DS-317990214-127.0.1.1-50010-1365505141363) service to localhost/127.0.0.1:8020 2013-04-11 16:26:16,276 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool BP-474150866-127.0.1.1-1365686732002 (storage id DS-317990214-127.0.1.1-50010-1365505141363) 2013-04-11 16:26:18,396 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode 2013-04-11 16:26:18,940 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0 2013-04-11 16:26:19,668 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: /************************************************** ********** SHUTDOWN_MSG: Shutting down DataNode at user-VirtualBox/127.0.1.1 ************************************************** **********/
Cause
When you initialize the NameNode either during a new installation or by formatting the NameNode for any reason, a new clusterID is created. You will see this exception If the datanodes are referring to a different clusterID that doesn’t match with the NameNode.
Solution
When the NameNode is formatted the DataNodes should also be formatted.
Find the location of the data directory in the DataNode where the HDFS blocks are stored. You can find the current location of the data directory dfs.datanode.data.dir of the cluster in hdfs-site.xml file under the property dfs.datanode.data.dir
<property> <name>dfs.datanode.data.dir</name> <value>file:/data/dfs/datanode/data</value> </property>
Once you have located the directory location, login in to all datanodes and remove the files and folders under the directory.
Start all the datanodes cnce the files and directories are cleaned up under the dfs.datanode.data.dir.