{"record":{"id":"3b351cc5907b4d13","repo":"apache/hadoop","slug":"cluster-ids-not-matched-dn-cid-clusterid-but-ns","errorCode":null,"errorMessage":"Cluster IDs not matched: dn cid={clusterId} but ns cid={nsCid}; bpid={bpid}","messagePattern":"Cluster IDs not matched: dn cid=(.+?) but ns cid=(.+?); bpid=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java","lineNumber":1444,"sourceCode":"\n    // Set configuration and dataDirs to reflect volume changes.\n    for (Iterator<StorageLocation> it = dataDirs.iterator(); it.hasNext(); ) {\n      StorageLocation loc = it.next();\n      if (storageLocations.contains(loc)) {\n        it.remove();\n      }\n    }\n    getConf().set(DFS_DATANODE_DATA_DIR_KEY, Joiner.on(\",\").join(dataDirs));\n\n    if (ioe != null) {\n      throw ioe;\n    }\n  }\n\n  private synchronized void setClusterId(final String nsCid, final String bpid\n      ) throws IOException {\n    if(clusterId != null && !clusterId.equals(nsCid)) {\n      throw new IOException (\"Cluster IDs not matched: dn cid=\" + clusterId \n          + \" but ns cid=\"+ nsCid + \"; bpid=\" + bpid);\n    }\n    // else\n    clusterId = nsCid;\n  }\n\n  /**\n   * Returns the hostname for this datanode. If the hostname is not\n   * explicitly configured in the given config, then it is determined\n   * via the DNS class.\n   *\n   * @param config configuration\n   * @return the hostname (NB: may not be a FQDN)\n   * @throws UnknownHostException if the dfs.datanode.dns.interface\n   *    option is used and the hostname can not be determined\n   */\n  private static String getHostName(Configuration config)\n      throws UnknownHostException {","sourceCodeStart":1426,"sourceCodeEnd":1462,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1426-L1462","documentation":"During block-pool handshake the DataNode compares its persisted clusterId (VERSION file under its storage dirs) with the clusterId the NameNode namespace announces (nsCid); setClusterId throws IOException when they differ, so the DN refuses to serve blocks for a foreign namespace. The bpid in the message identifies which block pool triggered the mismatch.","triggerScenarios":"NameNode was reformatted (fresh clusterID) while DataNode dirs keep the old VERSION; DN pointed at a different federation namespace via wrong dfs.nameservices / dfs.namenode.rpc-address; an HA pair rebuilt from scratch while DNs retain data from the old cluster.","commonSituations":"Re-running 'hdfs namenode -format' on a test cluster without cleaning DN dirs; swapping a test NN address for a prod one in DN configs; cloning VMs with stale storage directories.","solutions":["If the NN reformat was intentional, reset DN storage: stop the DN, remove or move aside the contents of its dfs.datanode.data.dir directories, then start it — it re-registers with the new clusterID","To preserve DN data, instead re-format the NN pinned to the old ID: hdfs namenode -format -clusterId <OLD_CLUSTER_ID>","If it is a config slip, fix dfs.nameservices / dfs.ha.namenodes / dfs.namenode.rpc-address so the DN talks to its own namespace"],"exampleFix":"# before: NN reformatted, DN still holds old VERSION -> Cluster IDs not matched\n# after\n$ hdfs --daemon stop datanode\n$ sudo -u hdfs rm -rf /data/dn*/current/*   # backup first if data matters\n$ hdfs --daemon start datanode","handlingStrategy":"validation","validationCode":"# Before starting/joining the DN, compare IDs:\n# NN:  curl -s http://nn:9870/jmx | jq '.beans[] | select(.name==\"Hadoop:service=NameNode,name=NameNodeStatus\") | .ClusterId'\n# DN:  grep clusterID /data/dn/current/VERSION\n# Abort the start if the strings differ.","typeGuard":null,"tryCatchPattern":"try {\n  dn.runDatanodeDaemon(); // setClusterId fires during BP handshake\n} catch (IOException e) {\n  if (e.getMessage().contains(\"Cluster IDs not matched\")) {\n    // decide: wipe DN storage (fresh NN) or reformat NN with -clusterId <old>\n  }\n}","preventionTips":["When reformatting a NameNode, pin the old ID: hdfs namenode -format -clusterId <OLD>","Automate DN dir cleanup together with NN reformat in test clusters","After cloning VMs, delete stale VERSION files before first start"],"tags":["hadoop","hdfs","datanode","cluster-id","formatting","startup"],"backgroundTag":"cluster-id-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}