{"record":{"id":"371a05b9004dc8c9","repo":"apache/hadoop","slug":"incompatible-namespaceids-in-namenode-namespac","errorCode":null,"errorMessage":"Incompatible namespaceIDs in {}: namenode namespaceID = {}; datanode namespaceID = {}","messagePattern":"Incompatible namespaceIDs in (.+?): namenode namespaceID = (.+?); datanode namespaceID = (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java","lineNumber":393,"sourceCode":"    if (startOpt == StartupOption.ROLLBACK && sd.getPreviousDir().exists()) {\n      Preconditions.checkState(!getTrashRootDir(sd).exists(),\n          sd.getPreviousDir() + \" and \" + getTrashRootDir(sd) + \" should not \" +\n          \" both be present.\");\n      doRollback(sd, nsInfo); // rollback if applicable\n    } else if (startOpt == StartupOption.ROLLBACK &&\n        !sd.getPreviousDir().exists()) {\n      // Restore all the files in the trash. The restored files are retained\n      // during rolling upgrade rollback. They are deleted during rolling\n      // upgrade downgrade.\n      int restored = restoreBlockFilesFromTrash(getTrashRootDir(sd));\n      LOG.info(\"Restored {} block files from trash.\", restored);\n    }\n    readProperties(sd);\n    checkVersionUpgradable(this.layoutVersion);\n    assert this.layoutVersion >= DataNodeLayoutVersion.getCurrentLayoutVersion()\n       : \"Future version is not allowed\";\n    if (getNamespaceID() != nsInfo.getNamespaceID()) {\n      throw new IOException(\"Incompatible namespaceIDs in \"\n          + sd.getRoot().getCanonicalPath() + \": namenode namespaceID = \"\n          + nsInfo.getNamespaceID() + \"; datanode namespaceID = \"\n          + getNamespaceID());\n    }\n    if (!blockpoolID.equals(nsInfo.getBlockPoolID())) {\n      throw new IOException(\"Incompatible blockpoolIDs in \"\n          + sd.getRoot().getCanonicalPath() + \": namenode blockpoolID = \"\n          + nsInfo.getBlockPoolID() + \"; datanode blockpoolID = \"\n          + blockpoolID);\n    }\n    if (this.layoutVersion == DataNodeLayoutVersion.getCurrentLayoutVersion()\n        && this.cTime == nsInfo.getCTime()) {\n      return false; // regular startup\n    }\n    if (this.layoutVersion > DataNodeLayoutVersion.getCurrentLayoutVersion()) {\n      int restored = restoreBlockFilesFromTrash(getTrashRootDir(sd));\n      LOG.info(\"Restored {} block files from trash \" +\n          \"before the layout upgrade. These blocks will be moved to \" +","sourceCodeStart":375,"sourceCodeEnd":411,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java#L375-L411","documentation":"During doTransition the namespaceID persisted in the DN's block pool storage differs from the NameNode's — this DN storage belongs to a different namespace, almost always because the NN was formatted (new namespaceID) while the DN kept its old data.","triggerScenarios":"hdfs namenode -format run while DataNodes kept their storage; DN pointed at the wrong cluster/nameservice; leftover dirs from a previous test deployment.","commonSituations":"Fresh-format of a dev/test cluster without cleaning DN dirs; misconfigured federation routing a DN to the wrong namespace.","solutions":["If the format was intentional, wipe the DN's block pool (or all storage) directories so it re-registers with the new namespace","Otherwise fix the DN's nameservice/NN addresses to point at the cluster this storage belongs to","If the format was accidental, restore the NN from backup — otherwise all DN blocks are orphaned"],"exampleFix":"# intentional reformat:\nrm -rf /dfs/dn/current/BP-*\nhdfs --daemon restart datanode","handlingStrategy":"validation","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (IOException e) {\n  if (e.getMessage().contains(\"Incompatible namespaceIDs\")) {\n    // expected after NN reformat: clean DN storage, restart to re-register\n    haltForOperator(\"wipe DN block pool dirs or fix NN address\");\n  }\n}","preventionTips":["Treat `hdfs namenode -format` as destructive for the whole cluster: plan DN storage cleanup with it","In test setups, script NN format + DN storage wipe together","Double-check the DN's nameservice config before first registration"],"tags":["hdfs","datanode","storage","namespaceid","identity-mismatch"],"backgroundTag":"cluster-identity-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}