apache/hadoop · error · IOException

Unexpected FS state: {curState} for storage directory: {root

Error message

Unexpected FS state: {curState} for storage directory: {rootPath}

What it means

Storage.completeChange(...) is a switch over the recoverable StorageStates returned by analyzeStorage (COMPLETE_UPGRADE, RECOVER_UPGRADE, COMPLETE_ROLLBACK, RECOVER_ROLLBACK, COMPLETE_FINALIZE, COMPLETE_CHECKPOINT, RECOVER_CHECKPOINT); its default branch throws IOException('Unexpected FS state: <curState> for storage directory: <rootPath>'). Reaching default means a state outside the recovery set arrived here — an internal invariant break, not a normal operator condition.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java:836

        LOG.info("Completing previous finalize for storage directory {}",
            rootPath);
        deleteAsync(getFinalizedTmp());
        return;
      case COMPLETE_CHECKPOINT: // mv lastcheckpoint.tmp -> previous.checkpoint
        LOG.info("Completing previous checkpoint for storage directory {}",
            rootPath);
        File prevCkptDir = getPreviousCheckpoint();
        deleteAsync(prevCkptDir);
        rename(getLastCheckpointTmp(), prevCkptDir);
        return;
      case RECOVER_CHECKPOINT:  // mv lastcheckpoint.tmp -> current
        LOG.info("Recovering storage directory {} from failed checkpoint",
            rootPath);
        deleteAsync(curDir);
        rename(getLastCheckpointTmp(), curDir);
        return;
      default:
        throw new IOException("Unexpected FS state: " + curState
            + " for storage directory: " + rootPath);
      }
    }

    /**
     * Rename the curDir to curDir.tmp and delete the curDir.tmp parallely.
     * @throws IOException
     */
    private void deleteAsync(File curDir) throws IOException {
      if (curDir.exists()) {
        File curTmp = new File(curDir.getParent(), curDir.getName() + ".tmp");
        if (curTmp.exists()) {
          deleteDir(curTmp);
        }
        rename(curDir, curTmp);
        new Thread("Async Delete Current.tmp") {
          public void run() {
            try {

View on GitHub (pinned to 2add963021)

Solutions

Verify every process touching that storage directory runs the identical Hadoop release (check jar versions on NN/DN/JN classpaths)
Capture the reported curState and correct the underlying directory state using the matching recovery procedure for that state
If unrecoverable, restore from checkpoint/format the directory
Report the case upstream including the state value and Hadoop versions involved, since this path indicates a bug

Defensive patterns

Strategy: try-catch

Validate before calling

Set<StorageState> recoverable = EnumSet.of(
    StorageState.COMPLETE_UPGRADE, StorageState.RECOVER_UPGRADE,
    StorageState.COMPLETE_ROLLBACK, StorageState.RECOVER_ROLLBACK,
    StorageState.COMPLETE_FINALIZE, StorageState.COMPLETE_CHECKPOINT,
    StorageState.RECOVER_CHECKPOINT);
StorageState st = sd.analyzeStorage(startOpt, storage, false);
if (!recoverable.contains(st)) {
  throw new IOException("State " + st + " is not recoverable for " + rootPath
      + "; check for mixed Hadoop versions on this storage directory");
}

Try / catch

try {
  sd.completeChange(startOpt, curState);
} catch (IOException e) {
  if (e.getMessage().startsWith("Unexpected FS state")) {
    // invariant break, not operator-fixable: capture state and versions
    LOG.error("Unhandled storage state {} in {}; Hadoop versions on classpath: {}",
        curState, rootPath, VersionInfo.getVersion());
    haltWithRunbook("Mixed Hadoop releases or patched Storage classes on one storage dir;");
  } else throw e;
}

Prevention

Run one Hadoop release per storage directory; never share or migrate dirs between mismatched versions mid-transition
Re-run 'hadoop version' on every node before starting upgrade/rollback transitions
Report occurrences upstream with the state value — this default branch indicates a code bug

When it happens

Trigger: A state such as NORMAL, NOT_FORMATTED or NON_EXISTENT reaching completeChange: caused by custom StorageDirectory subclasses returning unexpected states, or mixed Hadoop versions where one version's analyzeStorage and another's completeChange disagree on the same directory.

Common situations: Rolling upgrades with mismatched jars on the same storage directory; in-process patches that extend the StorageState enum without updating completeChange; experimental PROVIDED/mounted-storage configurations.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8187f9d9f52e1f51. Report an issue: GitHub.