{"record":{"id":"0ad9982b269a865b","repo":"apache/hadoop","slug":"one-and-only-one-directory-current-or-previous-mus","errorCode":null,"errorMessage":"one and only one directory current or previous must be present when removed.tmp exists.","messagePattern":"one and only one directory current or previous must be present when removed\\.tmp exists\\.","errorType":"exception","errorClass":"InconsistentFSStateException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java","lineNumber":772,"sourceCode":"          throw new InconsistentFSStateException(root,\n                                                 STORAGE_DIR_PREVIOUS + \" and \" + STORAGE_TMP_FINALIZED\n                                                 + \"cannot exist together.\");\n        return StorageState.COMPLETE_FINALIZE;\n      }\n\n      if (hasPreviousTmp) {\n        if (hasPrevious)\n          throw new InconsistentFSStateException(root,\n                                                 STORAGE_DIR_PREVIOUS + \" and \" + STORAGE_TMP_PREVIOUS\n                                                 + \" cannot exist together.\");\n        if (hasCurrent)\n          return StorageState.COMPLETE_UPGRADE;\n        return StorageState.RECOVER_UPGRADE;\n      }\n      \n      assert hasRemovedTmp : \"hasRemovedTmp must be true\";\n      if (!(hasCurrent ^ hasPrevious))\n        throw new InconsistentFSStateException(root,\n                                               \"one and only one directory \" + STORAGE_DIR_CURRENT \n                                               + \" or \" + STORAGE_DIR_PREVIOUS \n                                               + \" must be present when \" + STORAGE_TMP_REMOVED\n                                               + \" exists.\");\n      if (hasCurrent)\n        return StorageState.COMPLETE_ROLLBACK;\n      return StorageState.RECOVER_ROLLBACK;\n    }\n\n    /**\n     * Complete or recover storage state from previously failed transition.\n     * \n     * @param curState specifies what/how the state should be recovered\n     * @throws IOException\n     */\n    public void doRecover(StorageState curState) throws IOException {\n      File curDir = getCurrentDir();\n      if (curDir == null || root == null) {","sourceCodeStart":754,"sourceCodeEnd":790,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java#L754-L790","documentation":"When removed.tmp exists, a rollback is in flight (current/ was moved aside toward deletion). Recovery is only unambiguous when exactly one of current/ or previous/ remains — the XOR check !(hasCurrent ^ hasPrevious) throws InconsistentFSStateException('one and only one directory current or previous must be present when removed.tmp exists.') when both or neither survive. current/ present means COMPLETE_ROLLBACK; previous/ present means RECOVER_ROLLBACK.","triggerScenarios":"Crash during rollback where current/ was also lost or never restored, or where both current/ and previous/ exist alongside removed.tmp (e.g., partial manual recovery attempts that copied directories back).","commonSituations":"Interrupted rollbacks combined with operator intervention; restore-from-backup that recreated both dirs; disk corruption removing current/ mid-rollback.","solutions":["Stop the daemon and back up the entire storage directory before touching anything","Restore exactly one directory consistent with removed.tmp's contents: keep current/ to let COMPLETE_ROLLBACK finish deleting removed.tmp, or keep previous/ to let RECOVER_ROLLBACK undo the rollback","If neither side is recoverable, re-initialize the directory (format) and restore data from cluster-level replicas/backups"],"exampleFix":"# removed.tmp present, BOTH current/ and previous/ present\n# goal: finish rollback (keep current/)\nmv /dfs/nn/previous /backup/previous\n# restart namenode -> COMPLETE_ROLLBACK deletes removed.tmp -> NORMAL","handlingStrategy":"try-catch","validationCode":"File root = sd.getRoot();\nboolean hasCur = new File(root, \"current\").exists();\nboolean hasPrev = new File(root, \"previous\").exists();\nboolean hasRemoved = new File(root, \"removed.tmp\").exists();\nif (hasRemoved && hasCur == hasPrev) { // both or neither\n  throw new IOException(\"removed.tmp present but current/previous count is \"\n      + (hasCur ? 2 : 0) + \" in \" + root + \"; restore exactly one\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  daemon.start();\n} catch (InconsistentFSStateException e) {\n  if (e.getMessage().contains(\"removed.tmp\")) {\n    haltWithRunbook(\"Crashed rollback: keep current/ (complete rollback) or \"\n        + \"previous/ (undo rollback) — exactly one — then restart; back up first\");\n  } else throw e;\n}","preventionTips":["During rollback recovery, never recreate both current/ and previous/ manually","Back up the whole storage dir before choosing a rollback completion path","Prefer cluster-level replica recovery over hand-rebuilding a node's storage state"],"tags":["hdfs","storage","rollback","inconsistent-state","recovery"],"backgroundTag":"inconsistent-storage-state","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}