apache/hadoop · critical · java.io.IOException

File system image contains an old layout version {layoutVer

Error message

File system image contains an old layout version {layoutVersion}.
An upgrade to version {serviceLayoutVersion} is required.
Please restart NameNode with the "-rollingUpgrade started" option if a rolling upgrade is already started; or restart NameNode with the "-upgrade" option to start a new upgrade.

What it means

The on-disk fsimage layout version is older than the software's layout and older than Storage.LAST_PRE_UPGRADE_LAYOUT_VERSION-era rules permit for plain reading, while the startup option is not -upgrade/-upgradeOnly/-rollingUpgrade started. Hadoop forbids new binaries from silently adopting old metadata: moving the namespace to the new layout requires an explicit upgrade.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java:274


    int layoutVersion = storage.getLayoutVersion();
    if (startOpt == StartupOption.METADATAVERSION) {
      System.out.println("HDFS Image Version: " + layoutVersion);
      System.out.println("Software format version: " +
          storage.getServiceLayoutVersion());
      return false;
    }

    if (layoutVersion < Storage.LAST_PRE_UPGRADE_LAYOUT_VERSION) {
      NNStorage.checkVersionUpgradable(storage.getLayoutVersion());
    }
    if (startOpt != StartupOption.UPGRADE
        && startOpt != StartupOption.UPGRADEONLY
        && !RollingUpgradeStartupOption.STARTED.matches(startOpt)
        && layoutVersion < Storage.LAST_PRE_UPGRADE_LAYOUT_VERSION
        && layoutVersion != storage.getServiceLayoutVersion()) {
      throw new IOException(
          "\nFile system image contains an old layout version " 
          + storage.getLayoutVersion() + ".\nAn upgrade to version "
          + storage.getServiceLayoutVersion() + " is required.\n"
          + "Please restart NameNode with the \""
          + RollingUpgradeStartupOption.STARTED.getOptionString()
          + "\" option if a rolling upgrade is already started;"
          + " or restart NameNode with the \""
          + StartupOption.UPGRADE.getName() + "\" option to start"
          + " a new upgrade.");
    }
    
    storage.processStartupOptionsForUpgrade(startOpt, layoutVersion);

    // 2. Format unformatted dirs.
    for (Iterator<StorageDirectory> it = storage.dirIterator(); it.hasNext();) {
      StorageDirectory sd = it.next();
      StorageState curState = dataDirStates.get(sd);
      switch(curState) {

View on GitHub (pinned to 2add963021)

Solutions

If you intend to upgrade, restart with 'hdfs --daemon start namenode -upgrade' (then finalize later with 'hdfs dfsadmin -finalizeUpgrade').
If a rolling upgrade was already started on other nodes, restart with 'hdfs --daemon start namenode -rollingUpgrade started'.
If you did not intend to upgrade, run the previous Hadoop release binaries against this metadata.
If the metadata is older than the upgradable range, upgrade stepwise through intermediate releases (checkVersionUpgradable enforces that separately).

Example fix

# before
hdfs --daemon start namenode
# IOException: File system image contains an old layout version ...

# after
hdfs --daemon start namenode -upgrade
# or, for an in-progress rolling upgrade:
hdfs --daemon start namenode -rollingUpgrade started

Defensive patterns

Strategy: validation

Validate before calling

Properties props = new Properties();
try (InputStream is = Files.newInputStream(
        Paths.get(nameDir, "current", "VERSION"))) {
  props.load(is);
}
int lv = Integer.parseInt(props.getProperty("layoutVersion"));
int svc = HdfsServerConstants.NAMENODE_LAYOUT_VERSION;
if (lv != svc) {
  // decide before starting: -upgrade, -rollingUpgrade started, or old binaries
}

Prevention

Bake the '-upgrade' flag step into the release rollout runbook.
Record layoutVersion before and after every upgrade.
During rolling upgrades, always resume with '-rollingUpgrade started' until finalized.

When it happens

Trigger: Starting the NameNode normally after installing a newer Hadoop release over metadata formatted by an older release; resuming after a rolling upgrade was initiated elsewhere without passing -rollingUpgrade started; metadata volumes moved from an old cluster to new binaries.

Common situations: Upgrade runbook skips the -upgrade step; interrupted rolling upgrade resumed with a plain start; test clusters reusing old data dirs with new Hadoop tarballs.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d28b3676307b5f5f. Report an issue: GitHub.