apache/hadoop · critical · IOException

BUG: The stored LV = {} is newer than the supported LV = {}

Error message

BUG: The stored LV = {} is newer than the supported LV = {}

What it means

After readProperties and the upgrade branches, the only remaining possibility is a stored layoutVersion numerically smaller (i.e. newer) than DataNodeLayoutVersion.getCurrentLayoutVersion(). HDFS storage is forward-only, so this state - which checkVersionUpgradable during readProperties should normally have caught - is reported as a BUG: the running datanode binary is older than its on-disk metadata.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:777

    }

    // do upgrade
    if (this.layoutVersion > DataNodeLayoutVersion.getCurrentLayoutVersion()) {
      if (federationSupported) {
        // If the existing on-disk layout version supports federation,
        // simply update the properties.
        upgradeProperties(sd, conf);
      } else {
        doUpgradePreFederation(sd, nsInfo, callables, conf);
      }
      return true; // doUgrade already has written properties
    }
    
    // layoutVersion < DATANODE_LAYOUT_VERSION. I.e. stored layout version is newer
    // than the version supported by datanode. This should have been caught
    // in readProperties(), even if rollback was not carried out or somehow
    // failed.
    throw new IOException("BUG: The stored LV = " + this.getLayoutVersion()
        + " is newer than the supported LV = "
        + DataNodeLayoutVersion.getCurrentLayoutVersion());
  }

  /**
   * Upgrade from a pre-federation layout.
   * Move current storage into a backup directory,
   * and hardlink all its blocks into the new current directory.
   * 
   * Upgrade from pre-0.22 to 0.22 or later release e.g. 0.19/0.20/ => 0.22/0.23
   * <ul>
   * <li> If <SD>/previous exists then delete it </li>
   * <li> Rename <SD>/current to <SD>/previous.tmp </li>
   * <li>Create new <SD>/current/<bpid>/current directory<li>
   * <ul>
   * <li> Hard links for block files are created from <SD>/previous.tmp 
   * to <SD>/current/<bpid>/current </li>
   * <li> Saves new version file in <SD>/current/<bpid>/current directory </li>

View on GitHub (pinned to 2add963021)

Solutions

1. Upgrade the datanode software to at least the version that wrote the storage (match 'hadoop version' across the cluster)
2. Never downgrade HDFS metadata; to revert an upgrade use 'hdfs datanode -rollback' while binaries still match, restoring the previous snapshot
3. Verify the layoutVersion inside <dataDir>/current/VERSION against the active binary's supported layout

# before starting the DN, ensure the binary is not older than the stored layout
hadoop version
LV_STORED=$(grep '^layoutVersion=' $DATA_DIR/current/VERSION | cut -d= -f2)
# compare with the layout version supported by this release
# (org.apache.hadoop.hdfs.server.datanode.DataNodeLayoutVersion)
# if LV_STORED is numerically SMALLER than supported, this binary is too old -> upgrade first

catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("is newer than the supported LV")) {
    // on-disk metadata is newer than the binary: upgrade the Hadoop install; never downgrade storage
  } else { throw e; }
}

When it happens

Trigger: Starting an older Hadoop distribution against storage written by a newer version - e.g. rolling back the installed package after an upgrade, or a stale install path shadowing the new one - so stored LV < supported LV slips past the earlier checks.

Common situations: Package downgrade after a failed upgrade; alternates symlink still pointing at the old Hadoop release; test clusters flipping between versions without matching storage.

Related errors

• Unexpected version of storage directory {path}. Reported: {r
• Storage directory with location {} does not exist
• Storage directory is in use.
• Failed to remove %s: %s
• All specified directories have failed to load.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/07c58c5d0c913822. Report an issue: GitHub.