apache/hadoop · critical · IOException

BUG: The stored LV =

Error message

BUG: The stored LV = {} is newer than the supported LV = {}

What it means

After readProperties and the upgrade branches, the only remaining possibility is a stored layoutVersion numerically smaller (i.e. newer) than DataNodeLayoutVersion.getCurrentLayoutVersion(). HDFS storage is forward-only, so this state - which checkVersionUpgradable during readProperties should normally have caught - is reported as a BUG: the running datanode binary is older than its on-disk metadata.

Solutions

1. Upgrade the datanode software to at least the version that wrote the storage (match 'hadoop version' across the cluster)
2. Never downgrade HDFS metadata; to revert an upgrade use 'hdfs datanode -rollback' while binaries still match, restoring the previous snapshot
3. Verify the layoutVersion inside <dataDir>/current/VERSION against the active binary's supported layout

# before starting the DN, ensure the binary is not older than the stored layout
hadoop version
LV_STORED=$(grep '^layoutVersion=' $DATA_DIR/current/VERSION | cut -d= -f2)
# compare with the layout version supported by this release
# (org.apache.hadoop.hdfs.server.datanode.DataNodeLayoutVersion)
# if LV_STORED is numerically SMALLER than supported, this binary is too old -> upgrade first

catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("is newer than the supported LV")) {
    // on-disk metadata is newer than the binary: upgrade the Hadoop install; never downgrade storage
  } else { throw e; }
}

When it happens

Trigger: Starting an older Hadoop distribution against storage written by a newer version - e.g. rolling back the installed package after an upgrade, or a stale install path shadowing the new one - so stored LV < supported LV slips past the earlier checks.

Common situations: Package downgrade after a failed upgrade; alternates symlink still pointing at the old Hadoop release; test clusters flipping between versions without matching storage.

Related errors

• Unexpected version of storage directory
• All specified directories have failed to load.
• Cannot remove directory
• Detached directory is not empty. Please manually move each…
• Failed to remove

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/07c58c5d0c913822. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:777

    }

    // do upgrade
    if (this.layoutVersion > DataNodeLayoutVersion.getCurrentLayoutVersion()) {
      if (federationSupported) {
        // If the existing on-disk layout version supports federation,
        // simply update the properties.
        upgradeProperties(sd, conf);
      } else {
        doUpgradePreFederation(sd, nsInfo, callables, conf);
      }
      return true; // doUgrade already has written properties
    }
    
    // layoutVersion < DATANODE_LAYOUT_VERSION. I.e. stored layout version is newer
    // than the version supported by datanode. This should have been caught
    // in readProperties(), even if rollback was not carried out or somehow
    // failed.
    throw new IOException("BUG: The stored LV = " + this.getLayoutVersion()
        + " is newer than the supported LV = "
        + DataNodeLayoutVersion.getCurrentLayoutVersion());
  }

  /**
   * Upgrade from a pre-federation layout.
   * Move current storage into a backup directory,
   * and hardlink all its blocks into the new current directory.
   * 
   * Upgrade from pre-0.22 to 0.22 or later release e.g. 0.19/0.20/ => 0.22/0.23
   * <ul>
   * <li> If <SD>/previous exists then delete it </li>
   * <li> Rename <SD>/current to <SD>/previous.tmp </li>
   * <li>Create new <SD>/current/<bpid>/current directory<li>
   * <ul>
   * <li> Hard links for block files are created from <SD>/previous.tmp 
   * to <SD>/current/<bpid>/current </li>
   * <li> Saves new version file in <SD>/current/<bpid>/current directory </li>

View on GitHub (pinned to 2add963021)