{"record":{"id":"e17b691bdb2a3dff","repo":"apache/hadoop","slug":"incompatible-clusterids-in-namenode-clusterid","errorCode":null,"errorMessage":"Incompatible clusterIDs in {}: namenode clusterID = {}; datanode clusterID = {}","messagePattern":"Incompatible clusterIDs in (.+?): namenode clusterID = (.+?); datanode clusterID = (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java","lineNumber":750,"sourceCode":"        DataNodeLayoutVersion.getCurrentLayoutVersion() :\n        \"Future version is not allowed\";\n    \n    boolean federationSupported = \n      DataNodeLayoutVersion.supports(\n          LayoutVersion.Feature.FEDERATION, layoutVersion);\n    // For pre-federation version - validate the namespaceID\n    if (!federationSupported &&\n        getNamespaceID() != nsInfo.getNamespaceID()) {\n      throw new IOException(\"Incompatible namespaceIDs in \"\n          + sd.getRoot().getCanonicalPath() + \": namenode namespaceID = \"\n          + nsInfo.getNamespaceID() + \"; datanode namespaceID = \"\n          + getNamespaceID());\n    }\n    \n    // For version that supports federation, validate clusterID\n    if (federationSupported\n        && !getClusterID().equals(nsInfo.getClusterID())) {\n      throw new IOException(\"Incompatible clusterIDs in \"\n          + sd.getRoot().getCanonicalPath() + \": namenode clusterID = \"\n          + nsInfo.getClusterID() + \"; datanode clusterID = \" + getClusterID());\n    }\n\n    // regular start up.\n    if (this.layoutVersion == DataNodeLayoutVersion.getCurrentLayoutVersion()) {\n      createStorageID(sd, layoutVersion, conf);\n      return false; // need to write properties\n    }\n\n    // do upgrade\n    if (this.layoutVersion > DataNodeLayoutVersion.getCurrentLayoutVersion()) {\n      if (federationSupported) {\n        // If the existing on-disk layout version supports federation,\n        // simply update the properties.\n        upgradeProperties(sd, conf);\n      } else {\n        doUpgradePreFederation(sd, nsInfo, callables, conf);","sourceCodeStart":732,"sourceCodeEnd":768,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java#L732-L768","documentation":"With federation-era layouts, every DN storage dir records the clusterID of the NameNode that formatted it. During registration the DN compares it with the incoming NamespaceInfo clusterID; a mismatch aborts that directory with 'Incompatible clusterIDs in <dir>'. This is the classic signature of datanode storage left over from a previous/re-formatted namespace.","triggerScenarios":"NameNode was re-formatted (new clusterID) while dfs.datanode.data.dir still holds VERSION files with the old clusterID; or the DN points at a different cluster's NameNode (wrong nameservice / HA address).","commonSituations":"Running 'hdfs namenode -format' to reset a test cluster without clearing DN dirs; copying configs between environments so DNs contact another cluster's NN; partial re-format of an HA pair.","solutions":["If the re-format was intentional, also clear the DN storage on every datanode (delete the formatted dir contents) before restarting - this discards old replica metadata","If it was not intentional, restore the previous NN namespace (backup / HA standby) so the clusterID matches again","Verify dfs.nameservices and dfs.namenode.rpc-address point at the intended cluster","To keep IDs stable across test re-formats, format with an explicit -clusterId"],"exampleFix":"# before\nhdfs namenode -format          # new random clusterId; DNs still hold the old one\n\n# after (test-cluster reset)\nhdfs namenode -format -clusterId CID-mytest\n# ...or, if IDs already diverged: stop DN, clear its data dirs, restart DN","handlingStrategy":"validation","validationCode":"# before starting DNs, compare cluster IDs\nNN_CID=$(grep '^clusterID=' $NN_STORAGE/current/VERSION | cut -d= -f2)\nfor d in /data*/hdfs/dfs/data; do\n  DN_CID=$(grep '^clusterID=' $d/current/VERSION | cut -d= -f2 || echo none)\n  [ \"$DN_CID\" = \"$NN_CID\" ] || echo \"MISMATCH: $d has $DN_CID, NN has $NN_CID\"\ndone","typeGuard":null,"tryCatchPattern":"catch (IOException e) {\n  if (String.valueOf(e.getMessage()).contains(\"Incompatible clusterIDs\")) {\n    // stale DN storage vs re-formatted/different NN:\n    // intentional reset -> clear DN dirs and restart; accidental -> restore NN namespace\n  } else { throw e; }\n}","preventionTips":["Format test NNs with a fixed -clusterId and include DN-dir cleanup in the reset runbook","Double-check nameservice/NN addresses when copying configs between environments","Automate a clusterID consistency check between NN and DN VERSION files during provisioning"],"tags":["hadoop","hdfs","datanode","storage","clusterid","mismatch","reformat"],"backgroundTag":"cluster-id-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}