{"record":{"id":"52ec112c1bda13af","repo":"apache/hadoop","slug":"too-many-temporary-directories","errorCode":null,"errorMessage":"too many temporary directories.","messagePattern":"too many temporary directories\\.","errorType":"exception","errorClass":"InconsistentFSStateException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java","lineNumber":743,"sourceCode":"\n      if (!(hasPreviousTmp || hasRemovedTmp\n          || hasFinalizedTmp || hasCheckpointTmp)) {\n        // no temp dirs - no recovery\n        if (hasCurrent)\n          return StorageState.NORMAL;\n        if (hasPrevious)\n          throw new InconsistentFSStateException(root,\n                              \"version file in current directory is missing.\");\n        if (checkCurrentIsEmpty) {\n          checkEmptyCurrent();\n        }\n        return StorageState.NOT_FORMATTED;\n      }\n\n      if ((hasPreviousTmp?1:0) + (hasRemovedTmp?1:0)\n          + (hasFinalizedTmp?1:0) + (hasCheckpointTmp?1:0) > 1)\n        // more than one temp dirs\n        throw new InconsistentFSStateException(root,\n                                               \"too many temporary directories.\");\n\n      // # of temp dirs == 1 should either recover or complete a transition\n      if (hasCheckpointTmp) {\n        return hasCurrent ? StorageState.COMPLETE_CHECKPOINT\n                          : StorageState.RECOVER_CHECKPOINT;\n      }\n\n      if (hasFinalizedTmp) {\n        if (hasPrevious)\n          throw new InconsistentFSStateException(root,\n                                                 STORAGE_DIR_PREVIOUS + \" and \" + STORAGE_TMP_FINALIZED\n                                                 + \"cannot exist together.\");\n        return StorageState.COMPLETE_FINALIZE;\n      }\n\n      if (hasPreviousTmp) {\n        if (hasPrevious)","sourceCodeStart":725,"sourceCodeEnd":761,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java#L725-L761","documentation":"Storage.analyzeStorage tolerates at most one transition marker directory among previous.tmp, removed.tmp, finalized.tmp and lastcheckpoint.tmp; each marks one interrupted transition whose recovery the state machine knows how to complete. Counting more than one (the (x?1:0)+... > 1 check) is ambiguous and throws InconsistentFSStateException('too many temporary directories.') so no wrong transition is auto-chosen.","triggerScenarios":"Two transitions crashed in overlapping fashion — e.g., an upgrade leaving previous.tmp, then a botched manual rollback attempt leaving removed.tmp — so the daemon at boot sees two live markers.","commonSituations":"Repeated interrupted upgrades/finalizations; operators manually renaming directories to force recovery; restore of a backup over a directory that already had a temp marker.","solutions":["Stop all daemons sharing the storage directory and take a full backup of it","Identify the intended transition from timestamps and contents (e.g., previous.tmp = upgrade in flight; removed.tmp = rollback in flight)","Manually complete or delete the stale marker(s) so at most one remains (e.g., finish the rename the transition was performing, or remove the abandoned temp dir)","Restart the daemon and let analyzeStorage complete the single remaining transition"],"exampleFix":"# diagnosis: both previous.tmp and removed.tmp exist\nls /dfs/dn/current/../   # previous.tmp removed.tmp\n\n# recovery (daemons stopped, backup taken)\nmv /dfs/dn/removed.tmp /backup/   # keep upgrade marker only\n# restart datanode: COMPLETE_UPGRADE/RECOVER_UPGRADE proceeds","handlingStrategy":"try-catch","validationCode":"File root = sd.getRoot();\nString[] markers = {\"previous.tmp\", \"removed.tmp\", \"finalized.tmp\", \"lastcheckpoint.tmp\"};\nlong present = Arrays.stream(markers)\n    .filter(m -> new File(root, m).exists()).count();\nif (present > 1) {\n  throw new IOException(\"Ambiguous storage state: \" + present\n      + \" transition markers in \" + root + \"; manual recovery required\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  daemon.start();\n} catch (InconsistentFSStateException e) {\n  if (e.getMessage().contains(\"too many temporary directories\")) {\n    haltWithRunbook(\"Multiple interrupted transitions in \" + root\n        + \": stop daemons, back up, complete/remove stale *.tmp markers until one remains\");\n  } else throw e;\n}","preventionTips":["Never start a second transition (rollback/finalize) while recovering from a crashed one","After any crashed upgrade, inspect the storage dir for *.tmp markers before the next start","Back up storage dirs before manual marker surgery"],"tags":["hdfs","storage","upgrade","rollback","inconsistent-state","recovery"],"backgroundTag":"inconsistent-storage-state","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}