apache/hadoop · critical · IncorrectVersionException

storage directory " + sd.getRoot().getAbsolutePath()

Error message

storage directory " + sd.getRoot().getAbsolutePath()

What it means

During rolling-upgrade ROLLBACK startup, readProperties compares the layoutVersion in the directory's VERSION with the running software's service layout version; if storage was written by a newer release (lv > getServiceLayoutVersion()), IncorrectVersionException is thrown because old binaries cannot undo the newer layout changes. Downgrade via 'rollingupgrade rollback' is therefore not supported.

Solutions

1. Restart with the same or newer Hadoop version that wrote the storage, and finish the rollback from there.
2. If a real downgrade is required, use the supported downgrade path of the newer release ('rollingupgrade downgrade' with the new binaries), not old-binary rollback.
3. Verify the running binary with 'hdfs version' and compare it against the layoutVersion line in the dir's VERSION before choosing.

// Before rolling-upgrade rollback, compare on-disk layout with the running binary
int diskLv = Integer.parseInt(Files.readAllLines(versionFile).stream()
    .filter(l -> l.startsWith("layoutVersion=")).findFirst().orElseThrow()
    .split("=")[1]);
if (diskLv > HdfsServerConstants.NAMENODE_LAYOUT_VERSION) {
  throw new UnsupportedOperationException(
      "Storage layout " + diskLv + " is newer than this binary (" + HdfsServerConstants.NAMENODE_LAYOUT_VERSION
      + ") - rollback with the newer binaries or use rollingupgrade downgrade");
}

When it happens

Trigger: Running 'hdfs namenode -rollingupgrade rollback' with older-version binaries after the cluster already persisted a newer layoutVersion in VERSION - i.e. attempting a downgrade rollback after the upgrade was finalized or progressed too far.

Common situations: Aborting a failed rolling upgrade after NNs already upgraded; ops skipping the documented rollback-before-finalize order; PATH still pointing at old packages (rpm/alternatives leftovers) so the 'new' node actually runs old code.

Related errors

• Cannot rollback to a newer state. Datanode previous state…
• Cannot rollback to storage version
• Cannot rollback. None of the storage directories contain…
• Directory is in an inconsistent state: previous fs state…
• File system image contains an old layout version

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fe0cafdac6711dbf. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java:691

      String sbpid = props.getProperty("blockpoolID");
      setBlockPoolID(sd.getRoot(), sbpid);
    }
    setDeprecatedPropertiesForUpgrade(props);
  }

  void readProperties(StorageDirectory sd, StartupOption startupOption)
      throws IOException {
    Properties props = readPropertiesFile(sd.getVersionFile());
    if (props == null) {
      throw new IOException(
          "Properties not found  for storage directory " + sd);
    }
    if (HdfsServerConstants.RollingUpgradeStartupOption.ROLLBACK
        .matches(startupOption)) {
      int lv = Integer.parseInt(getProperty(props, sd, "layoutVersion"));
      if (lv > getServiceLayoutVersion()) {
        // we should not use a newer version for rollingUpgrade rollback
        throw new IncorrectVersionException(getServiceLayoutVersion(), lv,
            "storage directory " + sd.getRoot().getAbsolutePath());
      }
      props.setProperty("layoutVersion",
          Integer.toString(getServiceLayoutVersion()));
    }
    setFieldsFromProperties(props, sd);
  }

  /**
   * Pull any properties out of the VERSION file that are from older
   * versions of HDFS and only necessary during upgrade.
   */
  private void setDeprecatedPropertiesForUpgrade(Properties props) {
    deprecatedProperties = new HashMap<>();
    String md5 = props.getProperty(DEPRECATED_MESSAGE_DIGEST_PROPERTY);
    if (md5 != null) {
      deprecatedProperties.put(DEPRECATED_MESSAGE_DIGEST_PROPERTY, md5);
    }

View on GitHub (pinned to 2add963021)