apache/hadoop · critical · InconsistentFSStateException

cluster Id is incompatible with others.

Error message

cluster Id is incompatible with others.

What it means

StorageInfo.setClusterId runs when loading a storage directory on a federation-capable layout: the directory's clusterID must be empty (freshly formatted) or equal to the clusterID already loaded from earlier directories. Two different non-empty clusterIDs mean directories from different clusters are mixed in one daemon, which is unsafe, so InconsistentFSStateException is thrown.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/StorageInfo.java:208

          + " but StorageDirectory type=" + type);
    }
  }
  
  /** Validate and set ctime from {@link Properties}*/
  protected void setcTime(Properties props, StorageDirectory sd)
      throws InconsistentFSStateException {
    cTime = Long.parseLong(getProperty(props, sd, "cTime"));
  }

  /** Validate and set clusterId from {@link Properties}*/
  protected void setClusterId(Properties props, int layoutVersion,
      StorageDirectory sd) throws InconsistentFSStateException {
    // Set cluster ID in version that supports federation
    if (LayoutVersion.supports(getServiceLayoutFeatureMap(),
        Feature.FEDERATION, layoutVersion)) {
      String cid = getProperty(props, sd, "clusterID");
      if (!(clusterID.equals("") || cid.equals("") || clusterID.equals(cid))) {
        throw new InconsistentFSStateException(sd.getRoot(),
            "cluster Id is incompatible with others.");
      }
      clusterID = cid;
    }
  }
  
  /** Validate and set layout version from {@link Properties}*/
  protected void setLayoutVersion(Properties props, StorageDirectory sd)
      throws IncorrectVersionException, InconsistentFSStateException {
    int lv = Integer.parseInt(getProperty(props, sd, "layoutVersion"));
    if (lv < getServiceLayoutVersion()) { // future version
      throw new IncorrectVersionException(getServiceLayoutVersion(), lv,
          "storage directory " + sd.root.getAbsolutePath());
    }
    layoutVersion = lv;
  }
  
  /** Validate and set namespaceID version from {@link Properties}*/

View on GitHub (pinned to 2add963021)

Solutions

1. Identify the odd one out: cat <dir>/VERSION in every configured directory and compare clusterID
2. Wipe or reformat the mismatched directory with the running clusterID (hdfs namenode -format -clusterId <id> for NN; clear the block dirs for DN) - data in that dir is lost
3. Or simply remove the mismatched directory from the configuration list

Example fix

# before: dirs carry different clusterIDs
# after: reformat the stray dir against the running cluster
hdfs namenode -format -clusterId CID-20171105152119

Set<String> ids = new HashSet<>();
for (String d : configuredDirs) {
  Properties p = Storage.readPropertiesFile(new File(d, "VERSION"));
  String cid = p.getProperty("clusterID");
  if (cid != null && !cid.isEmpty()) ids.add(cid);
}
if (ids.size() > 1) throw new IOException("mixed clusterIDs in storage dirs: " + ids);

When it happens

Trigger: One of the configured storage directories was formatted against a different cluster (different clusterID than the rest); a leftover dir from a previous/test cluster left in dfs.namenode.name.dir or dfs.datanode.data.dir; a 2NN/BN bootstrap copied state from the wrong cluster.

Common situations: Docker/test clusters reusing volumes across reinstalls; adding a volume from a decommissioned cluster; reformatting the NameNode with a new clusterID while keeping old DataNode dirs.

Related errors

• Incompatible node types: storageType={storageType} but Stora
• namespaceID is incompatible with others.
• Unknown nameservice: {}
• Configuration has multiple addresses that match local node's
• Configuration dfs.namenode.rpc-address must be suffixed with

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/87d208878dbb840d. Report an issue: GitHub.