apache/hadoop · critical · IOException

Incompatible clusterIDs in {}: namenode clusterID = {}; data

Error message

Incompatible clusterIDs in {}: namenode clusterID = {}; datanode clusterID = {}

What it means

With federation-era layouts, every DN storage dir records the clusterID of the NameNode that formatted it. During registration the DN compares it with the incoming NamespaceInfo clusterID; a mismatch aborts that directory with 'Incompatible clusterIDs in <dir>'. This is the classic signature of datanode storage left over from a previous/re-formatted namespace.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:750

        DataNodeLayoutVersion.getCurrentLayoutVersion() :
        "Future version is not allowed";
    
    boolean federationSupported = 
      DataNodeLayoutVersion.supports(
          LayoutVersion.Feature.FEDERATION, layoutVersion);
    // For pre-federation version - validate the namespaceID
    if (!federationSupported &&
        getNamespaceID() != nsInfo.getNamespaceID()) {
      throw new IOException("Incompatible namespaceIDs in "
          + sd.getRoot().getCanonicalPath() + ": namenode namespaceID = "
          + nsInfo.getNamespaceID() + "; datanode namespaceID = "
          + getNamespaceID());
    }
    
    // For version that supports federation, validate clusterID
    if (federationSupported
        && !getClusterID().equals(nsInfo.getClusterID())) {
      throw new IOException("Incompatible clusterIDs in "
          + sd.getRoot().getCanonicalPath() + ": namenode clusterID = "
          + nsInfo.getClusterID() + "; datanode clusterID = " + getClusterID());
    }

    // regular start up.
    if (this.layoutVersion == DataNodeLayoutVersion.getCurrentLayoutVersion()) {
      createStorageID(sd, layoutVersion, conf);
      return false; // need to write properties
    }

    // do upgrade
    if (this.layoutVersion > DataNodeLayoutVersion.getCurrentLayoutVersion()) {
      if (federationSupported) {
        // If the existing on-disk layout version supports federation,
        // simply update the properties.
        upgradeProperties(sd, conf);
      } else {
        doUpgradePreFederation(sd, nsInfo, callables, conf);

View on GitHub (pinned to 2add963021)

Solutions

1. If the re-format was intentional, also clear the DN storage on every datanode (delete the formatted dir contents) before restarting - this discards old replica metadata
2. If it was not intentional, restore the previous NN namespace (backup / HA standby) so the clusterID matches again
3. Verify dfs.nameservices and dfs.namenode.rpc-address point at the intended cluster
4. To keep IDs stable across test re-formats, format with an explicit -clusterId

Example fix

# before
hdfs namenode -format          # new random clusterId; DNs still hold the old one

# after (test-cluster reset)
hdfs namenode -format -clusterId CID-mytest
# ...or, if IDs already diverged: stop DN, clear its data dirs, restart DN

# before starting DNs, compare cluster IDs
NN_CID=$(grep '^clusterID=' $NN_STORAGE/current/VERSION | cut -d= -f2)
for d in /data*/hdfs/dfs/data; do
  DN_CID=$(grep '^clusterID=' $d/current/VERSION | cut -d= -f2 || echo none)
  [ "$DN_CID" = "$NN_CID" ] || echo "MISMATCH: $d has $DN_CID, NN has $NN_CID"
done

catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("Incompatible clusterIDs")) {
    // stale DN storage vs re-formatted/different NN:
    // intentional reset -> clear DN dirs and restart; accidental -> restore NN namespace
  } else { throw e; }
}

When it happens

Trigger: NameNode was re-formatted (new clusterID) while dfs.datanode.data.dir still holds VERSION files with the old clusterID; or the DN points at a different cluster's NameNode (wrong nameservice / HA address).

Common situations: Running 'hdfs namenode -format' to reset a test cluster without clearing DN dirs; copying configs between environments so DNs contact another cluster's NN; partial re-format of an HA pair.

Related errors

• has incompatible storage Id.
• Root {}: DatanodeUuid={}, does not match {} from other Stora
• Incompatible namespaceIDs in {}: namenode namespaceID = {};
• " + idHelpText + " mismatch: previously connected to " + idH
• Storage directory with location {} does not exist

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e17b691bdb2a3dff. Report an issue: GitHub.