apache/hadoop · critical · IOException

Incompatible namespaceIDs in {}: namenode namespaceID = {};

Error message

Incompatible namespaceIDs in {}: namenode namespaceID = {}; datanode namespaceID = {}

What it means

For pre-federation storage layouts, DataStorage compares the namespaceID stored in the directory's VERSION with the NamespaceInfo sent by the NameNode at registration. A mismatch means the datanode storage belongs to a different (pre-federation) namespace, and that directory fails to load with this IOException.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:741

      createStorageID(sd, layoutVersion, conf);
      return false; // regular start up for PROVIDED storage directories
    }
    if (startOpt == StartupOption.ROLLBACK) {
      doRollback(sd, nsInfo); // rollback if applicable
    }
    readProperties(sd);
    checkVersionUpgradable(this.layoutVersion);
    assert this.layoutVersion >=
        DataNodeLayoutVersion.getCurrentLayoutVersion() :
        "Future version is not allowed";
    
    boolean federationSupported = 
      DataNodeLayoutVersion.supports(
          LayoutVersion.Feature.FEDERATION, layoutVersion);
    // For pre-federation version - validate the namespaceID
    if (!federationSupported &&
        getNamespaceID() != nsInfo.getNamespaceID()) {
      throw new IOException("Incompatible namespaceIDs in "
          + sd.getRoot().getCanonicalPath() + ": namenode namespaceID = "
          + nsInfo.getNamespaceID() + "; datanode namespaceID = "
          + getNamespaceID());
    }
    
    // For version that supports federation, validate clusterID
    if (federationSupported
        && !getClusterID().equals(nsInfo.getClusterID())) {
      throw new IOException("Incompatible clusterIDs in "
          + sd.getRoot().getCanonicalPath() + ": namenode clusterID = "
          + nsInfo.getClusterID() + "; datanode clusterID = " + getClusterID());
    }

    // regular start up.
    if (this.layoutVersion == DataNodeLayoutVersion.getCurrentLayoutVersion()) {
      createStorageID(sd, layoutVersion, conf);
      return false; // need to write properties
    }

View on GitHub (pinned to 2add963021)

Solutions

If the data is disposable, clear/reformat the datanode storage for that namespace and re-register
Point the datanode at the NameNode whose namespaceID matches the stored one
For genuine old-cluster upgrades, follow the documented pre-federation upgrade procedure instead of a plain restart

Defensive patterns

Strategy: try-catch

Try / catch

catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("Incompatible namespaceIDs")) {
    // pre-federation storage vs different NN namespace:
    // point DN at the right NN, or reformat/clear the storage deliberately
  } else { throw e; }
}

Prevention

On legacy clusters, always clear DN storage in the same maintenance step as an NN re-format
Pin test clusters to a fixed namespaceID so accidental re-formats keep IDs stable
Verify the NN's namespaceID (its VERSION file) matches DN storage before pointing DNs at it

When it happens

Trigger: Pre-federation DN storage connected to a re-formatted or different NameNode: nsInfo.getNamespaceID() != stored namespaceID when the stored layout version does not support the FEDERATION feature.

Common situations: Very old clusters (pre-0.22 layouts) after an NN re-format; test environments reusing legacy dirs against fresh NNs; historical upgrade paths skipping the documented procedure.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e828df0b2d57c089. Report an issue: GitHub.