apache/hadoop · error · InconsistentFSStateException

one and only one directory current or previous must be prese

Error message

one and only one directory current or previous must be present when removed.tmp exists.

What it means

When removed.tmp exists, a rollback is in flight (current/ was moved aside toward deletion). Recovery is only unambiguous when exactly one of current/ or previous/ remains — the XOR check !(hasCurrent ^ hasPrevious) throws InconsistentFSStateException('one and only one directory current or previous must be present when removed.tmp exists.') when both or neither survive. current/ present means COMPLETE_ROLLBACK; previous/ present means RECOVER_ROLLBACK.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java:772

          throw new InconsistentFSStateException(root,
                                                 STORAGE_DIR_PREVIOUS + " and " + STORAGE_TMP_FINALIZED
                                                 + "cannot exist together.");
        return StorageState.COMPLETE_FINALIZE;
      }

      if (hasPreviousTmp) {
        if (hasPrevious)
          throw new InconsistentFSStateException(root,
                                                 STORAGE_DIR_PREVIOUS + " and " + STORAGE_TMP_PREVIOUS
                                                 + " cannot exist together.");
        if (hasCurrent)
          return StorageState.COMPLETE_UPGRADE;
        return StorageState.RECOVER_UPGRADE;
      }
      
      assert hasRemovedTmp : "hasRemovedTmp must be true";
      if (!(hasCurrent ^ hasPrevious))
        throw new InconsistentFSStateException(root,
                                               "one and only one directory " + STORAGE_DIR_CURRENT 
                                               + " or " + STORAGE_DIR_PREVIOUS 
                                               + " must be present when " + STORAGE_TMP_REMOVED
                                               + " exists.");
      if (hasCurrent)
        return StorageState.COMPLETE_ROLLBACK;
      return StorageState.RECOVER_ROLLBACK;
    }

    /**
     * Complete or recover storage state from previously failed transition.
     * 
     * @param curState specifies what/how the state should be recovered
     * @throws IOException
     */
    public void doRecover(StorageState curState) throws IOException {
      File curDir = getCurrentDir();
      if (curDir == null || root == null) {

View on GitHub (pinned to 2add963021)

Solutions

Stop the daemon and back up the entire storage directory before touching anything
Restore exactly one directory consistent with removed.tmp's contents: keep current/ to let COMPLETE_ROLLBACK finish deleting removed.tmp, or keep previous/ to let RECOVER_ROLLBACK undo the rollback
If neither side is recoverable, re-initialize the directory (format) and restore data from cluster-level replicas/backups

Example fix

# removed.tmp present, BOTH current/ and previous/ present
# goal: finish rollback (keep current/)
mv /dfs/nn/previous /backup/previous
# restart namenode -> COMPLETE_ROLLBACK deletes removed.tmp -> NORMAL

Defensive patterns

Strategy: try-catch

Validate before calling

File root = sd.getRoot();
boolean hasCur = new File(root, "current").exists();
boolean hasPrev = new File(root, "previous").exists();
boolean hasRemoved = new File(root, "removed.tmp").exists();
if (hasRemoved && hasCur == hasPrev) { // both or neither
  throw new IOException("removed.tmp present but current/previous count is "
      + (hasCur ? 2 : 0) + " in " + root + "; restore exactly one");
}

Try / catch

try {
  daemon.start();
} catch (InconsistentFSStateException e) {
  if (e.getMessage().contains("removed.tmp")) {
    haltWithRunbook("Crashed rollback: keep current/ (complete rollback) or "
        + "previous/ (undo rollback) — exactly one — then restart; back up first");
  } else throw e;
}

Prevention

During rollback recovery, never recreate both current/ and previous/ manually
Back up the whole storage dir before choosing a rollback completion path
Prefer cluster-level replica recovery over hand-rebuilding a node's storage state

When it happens

Trigger: Crash during rollback where current/ was also lost or never restored, or where both current/ and previous/ exist alongside removed.tmp (e.g., partial manual recovery attempts that copied directories back).

Common situations: Interrupted rollbacks combined with operator intervention; restore-from-backup that recreated both dirs; disk corruption removing current/ mid-rollback.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0ad9982b269a865b. Report an issue: GitHub.