apache/hadoop · critical · IOException

Datanode state: LV = {} CTime = {} is newer than the namespa

Error message

Datanode state: LV = {} CTime = {} is newer than the namespace state: LV = {} CTime = {}

What it means

The DN's block pool state is newer than the namespace: layout is not upgradable (LV not above current) and cTime is greater than the NN's — typically the DN completed an upgrade while the NN was rolled back or restored to an older state. The DN shuts down because serving newer state against an older namespace would corrupt it.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java:421

    }
    if (this.layoutVersion == DataNodeLayoutVersion.getCurrentLayoutVersion()
        && this.cTime == nsInfo.getCTime()) {
      return false; // regular startup
    }
    if (this.layoutVersion > DataNodeLayoutVersion.getCurrentLayoutVersion()) {
      int restored = restoreBlockFilesFromTrash(getTrashRootDir(sd));
      LOG.info("Restored {} block files from trash " +
          "before the layout upgrade. These blocks will be moved to " +
          "the previous directory during the upgrade", restored);
    }
    if (this.layoutVersion > DataNodeLayoutVersion.getCurrentLayoutVersion()
        || this.cTime < nsInfo.getCTime()) {
      doUpgrade(sd, nsInfo, callables, conf); // upgrade
      return true;
    }
    // layoutVersion == LAYOUT_VERSION && this.cTime > nsInfo.cTime
    // must shutdown
    throw new IOException("Datanode state: LV = " + this.getLayoutVersion()
        + " CTime = " + this.getCTime()
        + " is newer than the namespace state: LV = "
        + nsInfo.getLayoutVersion() + " CTime = " + nsInfo.getCTime());
  }

  /**
   * Upgrade to any release after 0.22 (0.22 included) release
   * e.g. 0.22 =&gt; 0.23
   * Upgrade procedure is as follows:
   * <ol>
   * <li>If {@literal <SD>/current/<bpid>/previous} exists then delete it</li>
   * <li>Rename {@literal <SD>/current/<bpid>/current} to
   * {@literal <SD>/current/bpid/current/previous.tmp}</li>
   * <li>Create new {@literal <SD>current/<bpid>/current} directory</li>
   * <li>Hard links for block files are created from previous.tmp to current</li>
   * <li>Save new version file in current directory</li>
   * <li>Rename previous.tmp to previous</li>
   * </ol>

View on GitHub (pinned to 2add963021)

Solutions

1. Restart the DN with -rollback to revert to its previous snapshot
2. If the rollback was itself the mistake, restore the NN to the newer namespace state instead
3. As a last resort for disposable data, delete this BP storage and let the DN re-register

Example fix

# before
hdfs --daemon start datanode          # throws: newer than namespace

# after
hdfs --daemon start datanode -rollback

catch (IOException e) {
  if (e.getMessage().contains("is newer than the namespace state")) {
    // correct fix: restart this DN with -rollback
    haltForOperator("restart datanode with -rollback");
  }
}

When it happens

Trigger: Rolling upgrade rolled back on the NN but the DN was not rolled back; NN restored from an older image; DN storage copied from an upgraded cluster.

Common situations: Partial rollback procedures; downgrades attempted without rolling back every DN first.

Related errors

• Datanode CTime (={}) is not equal to namenode CTime (={})
• Incompatible node types: storageType={storageType} but Stora
• Storage directory for location {} and block pool id {} does
• BlockPoolSliceStorage.recoverTransitionRead: attempt to load
• file VERSION is invalid.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/dd2df98a62a02197. Report an issue: GitHub.