apache/hadoop · critical · InconsistentFSStateException

Root {}: DatanodeUuid={}, does not match {} from other Stora

Error message

Root {}: DatanodeUuid={}, does not match {} from other StorageDirectory.

What it means

All storage directories of one datanode must carry the same datanodeUuid (the datanodeUuid property in each VERSION). If a directory's UUID differs from the one already loaded from another directory, InconsistentFSStateException names the root and both UUIDs, and the DN refuses to start with mixed-identity directories.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:668

    String sid = sd.getStorageUuid();
    if (!(sid == null || sid.equals("") ||
          ssid.equals("") || sid.equals(ssid))) {
      throw new InconsistentFSStateException(sd.getRoot(),
          "has incompatible storage Id.");
    }

    if (sid == null) { // update id only if it was null
      sd.setStorageUuid(ssid);
    }

    // Update the datanode UUID if present.
    if (props.getProperty("datanodeUuid") != null) {
      String dnUuid = props.getProperty("datanodeUuid");

      if (getDatanodeUuid() == null) {
        setDatanodeUuid(dnUuid);
      } else if (getDatanodeUuid().compareTo(dnUuid) != 0) {
        throw new InconsistentFSStateException(sd.getRoot(),
            "Root " + sd.getRoot() + ": DatanodeUuid=" + dnUuid +
            ", does not match " + getDatanodeUuid() + " from other" +
            " StorageDirectory.");
      }
    }
  }

  @Override
  public boolean isPreUpgradableLayout(StorageDirectory sd) throws IOException {
    File oldF = new File(sd.getRoot(), "storage");
    if (!oldF.exists()) {
      return false;
    }
    // check the layout version inside the storage file
    // Lock and Read old storage file
    try (RandomAccessFile oldFile = new RandomAccessFile(oldF, "rws");
      FileLock oldLock = oldFile.getChannel().tryLock()) {
      if (null == oldLock) {

View on GitHub (pinned to 2add963021)

Solutions

1. Identify the foreign directory by comparing the two UUIDs in the message with the DN's expected DatanodeUuid (JMX / NN page)
2. Remove the mismatched dir from dfs.datanode.data.dir, or wipe/reformat it so all dirs share one UUID
3. Never share or copy DN storage dirs between datanodes; let the owning DN format new volumes in place

// Compare datanodeUuid across all data dirs before starting the DN
Set<String> uuids = new HashSet<>();
for (String dir : dataDirs) {
  Path v = Paths.get(dir, "current/VERSION");
  if (Files.exists(v)) {
    String line = Files.readAllLines(v).stream()
        .filter(l -> l.startsWith("datanodeUuid=")).findFirst().orElse(null);
    if (line != null) uuids.add(line.split("=", 2)[1]);
  }
}
if (uuids.size() > 1) throw new IOException("Mixed datanodeUuid in data dirs: " + uuids);

When it happens

Trigger: Two or more dfs.datanode.data.dir entries whose VERSION files carry different datanodeUuid values - one dir copied from another node, or a dir formatted by a different DN while the others are original.

Common situations: Cloned/reused disks moved between nodes; adding a volume that once belonged to another datanode; disaster-recovery restores that mix directories of different origins; manual dir swaps between hosts.

Related errors

• Failed to remove %s: %s
• All specified directories have failed to load.
• has incompatible storage Id.
• Incompatible namespaceIDs in {}: namenode namespaceID = {};
• Incompatible clusterIDs in {}: namenode clusterID = {}; data

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6cf258f8f30c2232. Report an issue: GitHub.