apache/hadoop · critical · IOException

Storage directories contain multiple layout versions: {layou

Error message

Storage directories contain multiple layout versions: {layoutVersions}

What it means

NNStorage collects the layoutVersion of every readable storage directory; if they disagree, the (dir, lv) pairs are echoed in the message and startup aborts. Mixed layout versions mean the directories were last written by different Hadoop releases, and the NN cannot safely pick one storage inspector for all of them.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java:1124

        continue;
      }
      readProperties(sd, startupOption); // sets layoutVersion
      int lv = getLayoutVersion();
      if (layoutVersion == null) {
        layoutVersion = lv;
      } else if (!layoutVersion.equals(lv)) {
        multipleLV = true;
      }
      layoutVersions.append("(").append(sd.getRoot()).append(", ").append(lv)
          .append(") ");
    }
    
    if (layoutVersion == null) {
      throw new IOException("No storage directories contained VERSION" +
          " information");
    }
    if (multipleLV) {            
      throw new IOException(
          "Storage directories contain multiple layout versions: "
              + layoutVersions);
    }
    // If the storage directories are with the new layout version
    // (ie edits_<txnid>) then use the new inspector, which will ignore
    // the old format dirs.
    FSImageStorageInspector inspector;
    if (NameNodeLayoutVersion.supports(
        LayoutVersion.Feature.TXID_BASED_LAYOUT, getLayoutVersion())) {
      inspector = new FSImageTransactionalStorageInspector(fileTypes);
    } else {
      inspector = new FSImagePreTransactionalStorageInspector();
    }
    
    inspectStorageDirs(inspector);
    return inspector;
  }

View on GitHub (pinned to 2add963021)

Solutions

Read the (dir, lv) pairs in the message and identify the minority-version directories.
Complete the upgrade with matching binaries: start with the newer version and run 'hdfs namenode -rollingupgrade finalize' so all dirs converge to one layoutVersion.
If the odd dirs are expendable, remove them from dfs.namenode.name.dir, restart, then clean them offline.
Never hand-edit layoutVersion - a wrong value makes the dir unparseable.

Defensive patterns

Strategy: validation

Validate before calling

// Pre-start check: all dirs must report one layoutVersion
Set<Integer> lvs = new HashSet<>();
for (String loc : conf.getTrimmedStrings("dfs.namenode.name.dir")) {
  lvs.add(Integer.parseInt(parseVersion(Paths.get(stripScheme(loc), "current", "VERSION"))
      .getOrDefault("layoutVersion", "-1")));
}
if (lvs.size() > 1) {
  throw new IllegalStateException("Mixed layout versions " + lvs + " - finish the upgrade/finalize before cold start");
}

Prevention

Finish (finalize) rolling upgrades before any cold restart so layouts converge.
Never copy name dirs between nodes running different Hadoop versions.
Include layoutVersion in the pre-start verification output for quick audits.

When it happens

Trigger: A dir written by an older or newer release is added to dfs.namenode.name.dir; an interrupted rolling upgrade left some dirs at the new layout and others at the old one; a partial rollback froze mid-transition.

Common situations: Rolling upgrade stopped midway and the cluster is cold-restarted; copying extra name dirs between nodes running different versions; downgrade attempts abandoned halfway.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/281efe1e7c405255. Report an issue: GitHub.