apache/hadoop · critical · InconsistentFSStateException

Cannot rollback to a newer state. Datanode previous state: L

Error message

Cannot rollback to a newer state.
Datanode previous state: LV = {} CTime = {} is newer than the namespace state: LV = {} CTime = {}

What it means

Thrown by DataStorage.rollback() as an InconsistentFSStateException when a DataNode is asked to roll back but its saved 'previous' snapshot is newer than the NameNode's current namespace state. Hadoop only permits rollback to a state that is older than or equal to the namespace: the previous layout version must be greater than or equal to (numerically, Hadoop layout versions count down as features are added) the current layout version, and the previous cTime must not exceed the namespace cTime. If either check fails, the DataNode refuses to roll back because rolling 'forward' the namespace state is not a supported operation.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:942

    if (!prevDir.exists()) {
      if (DataNodeLayoutVersion.supports(LayoutVersion.Feature.FEDERATION,
          DataNodeLayoutVersion.getCurrentLayoutVersion())) {
        readProperties(sd, DataNodeLayoutVersion.getCurrentLayoutVersion());
        writeProperties(sd);
        LOG.info("Layout version rolled back to {} for storage {}",
            DataNodeLayoutVersion.getCurrentLayoutVersion(), sd.getRoot());
      }
      return;
    }
    DataStorage prevInfo = new DataStorage();
    prevInfo.readPreviousVersionProperties(sd);

    // We allow rollback to a state, which is either consistent with
    // the namespace state or can be further upgraded to it.
    if (!(prevInfo.getLayoutVersion() >=
        DataNodeLayoutVersion.getCurrentLayoutVersion()
        && prevInfo.getCTime() <= nsInfo.getCTime())) {  // cannot rollback
      throw new InconsistentFSStateException(sd.getRoot(),
          "Cannot rollback to a newer state.\nDatanode previous state: LV = "
              + prevInfo.getLayoutVersion() + " CTime = " + prevInfo.getCTime()
              + " is newer than the namespace state: LV = "
              + DataNodeLayoutVersion.getCurrentLayoutVersion() + " CTime = "
              + nsInfo.getCTime());
    }
    LOG.info("Rolling back storage directory {}.\n   target LV = {}; target "
            + "CTime = {}", sd.getRoot(),
        DataNodeLayoutVersion.getCurrentLayoutVersion(), nsInfo.getCTime());
    File tmpDir = sd.getRemovedTmp();
    assert !tmpDir.exists() : "removed.tmp directory must not exist.";
    // rename current to tmp
    File curDir = sd.getCurrentDir();
    assert curDir.exists() : "Current directory must exist.";
    rename(curDir, tmpDir);
    // rename previous to current
    rename(prevDir, curDir);
    // delete tmp dir

View on GitHub (pinned to 2add963021)

Solutions

1. Roll the NameNode back first ('hdfs namenode -rollback') and verify its LV/CTime, then restart DataNodes with '-rollback', so the namespace is not older than the DataNode's previous state
2. Verify the whole cluster runs one Hadoop version (hdfs version on each node) so DataNode previous-state layout version is not newer than the NameNode's
3. If the NameNode state is fresh (reformatted or newer cTime needed), perform a full upgrade with '-upgrade' instead of a rollback
4. As a last resort on a throwaway cluster, clear the DataNode storage directory (or the previous/ and current/ dirs) and let the node re-register and re-replicate its blocks

Example fix

# before: NN left on older state, DN previous state is newer -> InconsistentFSStateException
hdfs --daemon stop datanode
hdfs --daemon start datanode -rollback

# after: roll NameNode back first so namespace >= DN previous state, then DN
hdfs --daemon stop namenode
hdfs namenode -rollback
hdfs --daemon start namenode
hdfs --daemon start datanode -rollback

// Before rollback, compare DN previous-state LV/CTime with the NN state
// (read from previous/<role>.md or VERSION files after upgrade):
Properties prev = loadVersionFile(new File(dnDir, "current/BP-*/previous/VERSION"));
int prevLV = Integer.parseInt(prev.getProperty("layoutVersion"));
long prevCT = Long.parseLong(prev.getProperty("cTime"));
if (!(prevLV >= currentLayoutVersion /* NN's LV */
      && prevCT <= nnCTime)) {
  throw new IllegalStateException(
      "DN previous state is newer than NN namespace - rollback NN first or upgrade instead");
}
// safe to: hdfs --daemon start datanode -rollback

try {
  storage.rollback(sd, nsInfo);
} catch (InconsistentFSStateException e) {
  // abort rollback for this volume; report LV/CTime pair from the message
  // do NOT delete previous/ - operator must align NN/DN versions first
  log.error("Rollback refused for {}: {}", sd.getRoot(), e.getMessage());
}

When it happens

Trigger: Running 'hdfs namenode -rollback' / starting the DataNode with rollback when the DataNode's previous/ directory was written by a newer Hadoop release (lower layout version number) or a later namespace creation time than the running NameNode's state. Concretely triggered when !(prevInfo.getLayoutVersion() >= DataNodeLayoutVersion.getCurrentLayoutVersion() && prevInfo.getCTime() <= nsInfo.getCTime()).

Common situations: NameNode was rolled back to an older image but some DataNodes had already upgraded past it; mixed Hadoop versions across the cluster after a partial upgrade/downgrade; NameNode re-formatted (fresh cTime) while DataNodes still hold upgraded state; a rollback attempted after the previous checkpoint was itself the product of a newer release.

Related errors

• storage directory " + sd.getRoot().getAbsolutePath()
• Cannot rollback to storage version {prevStorage.getLayoutVer
• too many temporary directories.
• *********** Upgrade is not supported from this older versio
• Datanode CTime (={}) is not equal to namenode CTime (={})

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/cd86ad67cc574c51. Report an issue: GitHub.