apache/hadoop · critical · IOException

Incompatible namespaceIDs in {}: namenode namespaceID = {};

Error message

Incompatible namespaceIDs in {}: namenode namespaceID = {}; datanode namespaceID = {}

What it means

During doTransition the namespaceID persisted in the DN's block pool storage differs from the NameNode's — this DN storage belongs to a different namespace, almost always because the NN was formatted (new namespaceID) while the DN kept its old data.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java:393

    if (startOpt == StartupOption.ROLLBACK && sd.getPreviousDir().exists()) {
      Preconditions.checkState(!getTrashRootDir(sd).exists(),
          sd.getPreviousDir() + " and " + getTrashRootDir(sd) + " should not " +
          " both be present.");
      doRollback(sd, nsInfo); // rollback if applicable
    } else if (startOpt == StartupOption.ROLLBACK &&
        !sd.getPreviousDir().exists()) {
      // Restore all the files in the trash. The restored files are retained
      // during rolling upgrade rollback. They are deleted during rolling
      // upgrade downgrade.
      int restored = restoreBlockFilesFromTrash(getTrashRootDir(sd));
      LOG.info("Restored {} block files from trash.", restored);
    }
    readProperties(sd);
    checkVersionUpgradable(this.layoutVersion);
    assert this.layoutVersion >= DataNodeLayoutVersion.getCurrentLayoutVersion()
       : "Future version is not allowed";
    if (getNamespaceID() != nsInfo.getNamespaceID()) {
      throw new IOException("Incompatible namespaceIDs in "
          + sd.getRoot().getCanonicalPath() + ": namenode namespaceID = "
          + nsInfo.getNamespaceID() + "; datanode namespaceID = "
          + getNamespaceID());
    }
    if (!blockpoolID.equals(nsInfo.getBlockPoolID())) {
      throw new IOException("Incompatible blockpoolIDs in "
          + sd.getRoot().getCanonicalPath() + ": namenode blockpoolID = "
          + nsInfo.getBlockPoolID() + "; datanode blockpoolID = "
          + blockpoolID);
    }
    if (this.layoutVersion == DataNodeLayoutVersion.getCurrentLayoutVersion()
        && this.cTime == nsInfo.getCTime()) {
      return false; // regular startup
    }
    if (this.layoutVersion > DataNodeLayoutVersion.getCurrentLayoutVersion()) {
      int restored = restoreBlockFilesFromTrash(getTrashRootDir(sd));
      LOG.info("Restored {} block files from trash " +
          "before the layout upgrade. These blocks will be moved to " +

View on GitHub (pinned to 2add963021)

Solutions

1. If the format was intentional, wipe the DN's block pool (or all storage) directories so it re-registers with the new namespace
2. Otherwise fix the DN's nameservice/NN addresses to point at the cluster this storage belongs to
3. If the format was accidental, restore the NN from backup — otherwise all DN blocks are orphaned

Example fix

# intentional reformat:
rm -rf /dfs/dn/current/BP-*
hdfs --daemon restart datanode

catch (IOException e) {
  if (e.getMessage().contains("Incompatible namespaceIDs")) {
    // expected after NN reformat: clean DN storage, restart to re-register
    haltForOperator("wipe DN block pool dirs or fix NN address");
  }
}

When it happens

Trigger: hdfs namenode -format run while DataNodes kept their storage; DN pointed at the wrong cluster/nameservice; leftover dirs from a previous test deployment.

Common situations: Fresh-format of a dev/test cluster without cleaning DN dirs; misconfigured federation routing a DN to the wrong namespace.

Related errors

• Unexpected blockpoolID {}. Expected {}
• Incompatible blockpoolIDs in {}: namenode blockpoolID = {};
• Incompatible namespaceIDs in {}: namenode namespaceID = {};
• Incompatible node types: storageType={storageType} but Stora
• " + idHelpText + " mismatch: previously connected to " + idH

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/371a05b9004dc8c9. Report an issue: GitHub.