What causes "too many temporary directories." in apache/hadoop?

Stop all daemons sharing the storage directory and take a full backup of it

apache/hadoop · error · InconsistentFSStateException

too many temporary directories.

Error message

too many temporary directories.

What it means

Storage.analyzeStorage tolerates at most one transition marker directory among previous.tmp, removed.tmp, finalized.tmp and lastcheckpoint.tmp; each marks one interrupted transition whose recovery the state machine knows how to complete. Counting more than one (the (x?1:0)+... > 1 check) is ambiguous and throws InconsistentFSStateException('too many temporary directories.') so no wrong transition is auto-chosen.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java:743

      if (!(hasPreviousTmp || hasRemovedTmp
          || hasFinalizedTmp || hasCheckpointTmp)) {
        // no temp dirs - no recovery
        if (hasCurrent)
          return StorageState.NORMAL;
        if (hasPrevious)
          throw new InconsistentFSStateException(root,
                              "version file in current directory is missing.");
        if (checkCurrentIsEmpty) {
          checkEmptyCurrent();
        }
        return StorageState.NOT_FORMATTED;
      }

      if ((hasPreviousTmp?1:0) + (hasRemovedTmp?1:0)
          + (hasFinalizedTmp?1:0) + (hasCheckpointTmp?1:0) > 1)
        // more than one temp dirs
        throw new InconsistentFSStateException(root,
                                               "too many temporary directories.");

      // # of temp dirs == 1 should either recover or complete a transition
      if (hasCheckpointTmp) {
        return hasCurrent ? StorageState.COMPLETE_CHECKPOINT
                          : StorageState.RECOVER_CHECKPOINT;
      }

      if (hasFinalizedTmp) {
        if (hasPrevious)
          throw new InconsistentFSStateException(root,
                                                 STORAGE_DIR_PREVIOUS + " and " + STORAGE_TMP_FINALIZED
                                                 + "cannot exist together.");
        return StorageState.COMPLETE_FINALIZE;
      }

      if (hasPreviousTmp) {
        if (hasPrevious)

View on GitHub (pinned to 2add963021)

Solutions

Stop all daemons sharing the storage directory and take a full backup of it
Identify the intended transition from timestamps and contents (e.g., previous.tmp = upgrade in flight; removed.tmp = rollback in flight)
Manually complete or delete the stale marker(s) so at most one remains (e.g., finish the rename the transition was performing, or remove the abandoned temp dir)
Restart the daemon and let analyzeStorage complete the single remaining transition

Example fix

# diagnosis: both previous.tmp and removed.tmp exist
ls /dfs/dn/current/../   # previous.tmp removed.tmp

# recovery (daemons stopped, backup taken)
mv /dfs/dn/removed.tmp /backup/   # keep upgrade marker only
# restart datanode: COMPLETE_UPGRADE/RECOVER_UPGRADE proceeds

Defensive patterns

Strategy: try-catch

Validate before calling

File root = sd.getRoot();
String[] markers = {"previous.tmp", "removed.tmp", "finalized.tmp", "lastcheckpoint.tmp"};
long present = Arrays.stream(markers)
    .filter(m -> new File(root, m).exists()).count();
if (present > 1) {
  throw new IOException("Ambiguous storage state: " + present
      + " transition markers in " + root + "; manual recovery required");
}

Try / catch

try {
  daemon.start();
} catch (InconsistentFSStateException e) {
  if (e.getMessage().contains("too many temporary directories")) {
    haltWithRunbook("Multiple interrupted transitions in " + root
        + ": stop daemons, back up, complete/remove stale *.tmp markers until one remains");
  } else throw e;
}

Prevention

Never start a second transition (rollback/finalize) while recovering from a crashed one
After any crashed upgrade, inspect the storage dir for *.tmp markers before the next start
Back up storage dirs before manual marker surgery

When it happens

Trigger: Two transitions crashed in overlapping fashion — e.g., an upgrade leaving previous.tmp, then a botched manual rollback attempt leaving removed.tmp — so the daemon at boot sees two live markers.

Common situations: Repeated interrupted upgrades/finalizations; operators manually renaming directories to force recovery; restore of a backup over a directory that already had a temp marker.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/52ec112c1bda13af. Report an issue: GitHub.