apache/hadoop · error · InconsistentFSStateException

Cannot rollback to a newer state. Datanode previous state: L

Error message

Cannot rollback to a newer state.
Datanode previous state: LV = {} CTime = {} is newer than the namespace state: LV = {} CTime = {}

What it means

Error "Cannot rollback to a newer state. Datanode previous state: LV = {} CTime = {} is newer than the namespace state: LV = {} CTime = {}" thrown in apache/hadoop.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java:621

  void doRollback(StorageDirectory bpSd, NamespaceInfo nsInfo)
      throws IOException {
    File prevDir = bpSd.getPreviousDir();
    // regular startup if previous dir does not exist
    if (prevDir == null || !prevDir.exists()) {
      return;
    }
    // read attributes out of the VERSION file of previous directory
    BlockPoolSliceStorage prevInfo = new BlockPoolSliceStorage();
    prevInfo.readPreviousVersionProperties(bpSd);

    // We allow rollback to a state, which is either consistent with
    // the namespace state or can be further upgraded to it.
    // In another word, we can only roll back when ( storedLV >= software LV)
    // && ( DN.previousCTime <= NN.ctime)
    if (!(prevInfo.getLayoutVersion() >=
        DataNodeLayoutVersion.getCurrentLayoutVersion() &&
        prevInfo.getCTime() <= nsInfo.getCTime())) { // cannot rollback
      throw new InconsistentFSStateException(bpSd.getRoot(),
          "Cannot rollback to a newer state.\nDatanode previous state: LV = "
              + prevInfo.getLayoutVersion() + " CTime = " + prevInfo.getCTime()
              + " is newer than the namespace state: LV = "
              + DataNodeLayoutVersion.getCurrentLayoutVersion() + " CTime = "
              + nsInfo.getCTime());
    }

    LOG.info("Rolling back storage directory {}.\n   target LV = {}; target "
            + "CTime = {}", bpSd.getRoot(), nsInfo.getLayoutVersion(),
        nsInfo.getCTime());
    File tmpDir = bpSd.getRemovedTmp();
    assert !tmpDir.exists() : "removed.tmp directory must not exist.";
    // 1. rename current to tmp
    File curDir = bpSd.getCurrentDir();
    assert curDir.exists() : "Current directory must exist.";
    rename(curDir, tmpDir);
    
    // 2. rename previous to current

View on GitHub (pinned to 2add963021)

Solutions

1. Do not attempt a rollback (-rollback) when the DataNode's stored layout version or CTime is newer than the NameNode's namespace state; rollback is only supported to an older state.
2. Verify the DataNode and NameNode software versions match, and check the storage directory VERSION files (layoutVersion and cTime) on both sides to confirm which side was upgraded first.

When it happens

Trigger: Rolling back an HDFS upgrade when the DataNode storage was already formatted or finalized by a newer software version than the NameNode, or when the VERSION cTime on the DataNode is ahead of the namespace state.

Common situations: Rollback requested while the DataNode storage layout version or cTime is newer than the NameNode namespace state. Prevent by verifying LV/CTime in each storage directory's VERSION file before running rollback.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6965d95a9c4c518b. Report an issue: GitHub.