{"record":{"id":"8187f9d9f52e1f51","repo":"apache/hadoop","slug":"unexpected-fs-state-curstate-for-storage-direct","errorCode":null,"errorMessage":"Unexpected FS state: {curState} for storage directory: {rootPath}","messagePattern":"Unexpected FS state: (.+?) for storage directory: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java","lineNumber":836,"sourceCode":"        LOG.info(\"Completing previous finalize for storage directory {}\",\n            rootPath);\n        deleteAsync(getFinalizedTmp());\n        return;\n      case COMPLETE_CHECKPOINT: // mv lastcheckpoint.tmp -> previous.checkpoint\n        LOG.info(\"Completing previous checkpoint for storage directory {}\",\n            rootPath);\n        File prevCkptDir = getPreviousCheckpoint();\n        deleteAsync(prevCkptDir);\n        rename(getLastCheckpointTmp(), prevCkptDir);\n        return;\n      case RECOVER_CHECKPOINT:  // mv lastcheckpoint.tmp -> current\n        LOG.info(\"Recovering storage directory {} from failed checkpoint\",\n            rootPath);\n        deleteAsync(curDir);\n        rename(getLastCheckpointTmp(), curDir);\n        return;\n      default:\n        throw new IOException(\"Unexpected FS state: \" + curState\n            + \" for storage directory: \" + rootPath);\n      }\n    }\n\n    /**\n     * Rename the curDir to curDir.tmp and delete the curDir.tmp parallely.\n     * @throws IOException\n     */\n    private void deleteAsync(File curDir) throws IOException {\n      if (curDir.exists()) {\n        File curTmp = new File(curDir.getParent(), curDir.getName() + \".tmp\");\n        if (curTmp.exists()) {\n          deleteDir(curTmp);\n        }\n        rename(curDir, curTmp);\n        new Thread(\"Async Delete Current.tmp\") {\n          public void run() {\n            try {","sourceCodeStart":818,"sourceCodeEnd":854,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java#L818-L854","documentation":"Storage.completeChange(...) is a switch over the recoverable StorageStates returned by analyzeStorage (COMPLETE_UPGRADE, RECOVER_UPGRADE, COMPLETE_ROLLBACK, RECOVER_ROLLBACK, COMPLETE_FINALIZE, COMPLETE_CHECKPOINT, RECOVER_CHECKPOINT); its default branch throws IOException('Unexpected FS state: <curState> for storage directory: <rootPath>'). Reaching default means a state outside the recovery set arrived here — an internal invariant break, not a normal operator condition.","triggerScenarios":"A state such as NORMAL, NOT_FORMATTED or NON_EXISTENT reaching completeChange: caused by custom StorageDirectory subclasses returning unexpected states, or mixed Hadoop versions where one version's analyzeStorage and another's completeChange disagree on the same directory.","commonSituations":"Rolling upgrades with mismatched jars on the same storage directory; in-process patches that extend the StorageState enum without updating completeChange; experimental PROVIDED/mounted-storage configurations.","solutions":["Verify every process touching that storage directory runs the identical Hadoop release (check jar versions on NN/DN/JN classpaths)","Capture the reported curState and correct the underlying directory state using the matching recovery procedure for that state","If unrecoverable, restore from checkpoint/format the directory","Report the case upstream including the state value and Hadoop versions involved, since this path indicates a bug"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"Set<StorageState> recoverable = EnumSet.of(\n    StorageState.COMPLETE_UPGRADE, StorageState.RECOVER_UPGRADE,\n    StorageState.COMPLETE_ROLLBACK, StorageState.RECOVER_ROLLBACK,\n    StorageState.COMPLETE_FINALIZE, StorageState.COMPLETE_CHECKPOINT,\n    StorageState.RECOVER_CHECKPOINT);\nStorageState st = sd.analyzeStorage(startOpt, storage, false);\nif (!recoverable.contains(st)) {\n  throw new IOException(\"State \" + st + \" is not recoverable for \" + rootPath\n      + \"; check for mixed Hadoop versions on this storage directory\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  sd.completeChange(startOpt, curState);\n} catch (IOException e) {\n  if (e.getMessage().startsWith(\"Unexpected FS state\")) {\n    // invariant break, not operator-fixable: capture state and versions\n    LOG.error(\"Unhandled storage state {} in {}; Hadoop versions on classpath: {}\",\n        curState, rootPath, VersionInfo.getVersion());\n    haltWithRunbook(\"Mixed Hadoop releases or patched Storage classes on one storage dir;\");\n  } else throw e;\n}","preventionTips":["Run one Hadoop release per storage directory; never share or migrate dirs between mismatched versions mid-transition","Re-run 'hadoop version' on every node before starting upgrade/rollback transitions","Report occurrences upstream with the state value — this default branch indicates a code bug"],"tags":["hdfs","storage","recovery","internal-error","version-mismatch","state-machine"],"backgroundTag":"unhandled-storage-state","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}