{"record":{"id":"cd86ad67cc574c51","repo":"apache/hadoop","slug":"cannot-rollback-to-a-newer-state-datanode-previou-cd86ad","errorCode":null,"errorMessage":"Cannot rollback to a newer state.\nDatanode previous state: LV = {} CTime = {} is newer than the namespace state: LV = {} CTime = {}","messagePattern":"Cannot rollback to a newer state\\.\nDatanode previous state: LV = (.+?) CTime = (.+?) is newer than the namespace state: LV = (.+?) CTime = (.+?)","errorType":"exception","errorClass":"InconsistentFSStateException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java","lineNumber":942,"sourceCode":"    if (!prevDir.exists()) {\n      if (DataNodeLayoutVersion.supports(LayoutVersion.Feature.FEDERATION,\n          DataNodeLayoutVersion.getCurrentLayoutVersion())) {\n        readProperties(sd, DataNodeLayoutVersion.getCurrentLayoutVersion());\n        writeProperties(sd);\n        LOG.info(\"Layout version rolled back to {} for storage {}\",\n            DataNodeLayoutVersion.getCurrentLayoutVersion(), sd.getRoot());\n      }\n      return;\n    }\n    DataStorage prevInfo = new DataStorage();\n    prevInfo.readPreviousVersionProperties(sd);\n\n    // We allow rollback to a state, which is either consistent with\n    // the namespace state or can be further upgraded to it.\n    if (!(prevInfo.getLayoutVersion() >=\n        DataNodeLayoutVersion.getCurrentLayoutVersion()\n        && prevInfo.getCTime() <= nsInfo.getCTime())) {  // cannot rollback\n      throw new InconsistentFSStateException(sd.getRoot(),\n          \"Cannot rollback to a newer state.\\nDatanode previous state: LV = \"\n              + prevInfo.getLayoutVersion() + \" CTime = \" + prevInfo.getCTime()\n              + \" is newer than the namespace state: LV = \"\n              + DataNodeLayoutVersion.getCurrentLayoutVersion() + \" CTime = \"\n              + nsInfo.getCTime());\n    }\n    LOG.info(\"Rolling back storage directory {}.\\n   target LV = {}; target \"\n            + \"CTime = {}\", sd.getRoot(),\n        DataNodeLayoutVersion.getCurrentLayoutVersion(), nsInfo.getCTime());\n    File tmpDir = sd.getRemovedTmp();\n    assert !tmpDir.exists() : \"removed.tmp directory must not exist.\";\n    // rename current to tmp\n    File curDir = sd.getCurrentDir();\n    assert curDir.exists() : \"Current directory must exist.\";\n    rename(curDir, tmpDir);\n    // rename previous to current\n    rename(prevDir, curDir);\n    // delete tmp dir","sourceCodeStart":924,"sourceCodeEnd":960,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java#L924-L960","documentation":"Thrown by DataStorage.rollback() as an InconsistentFSStateException when a DataNode is asked to roll back but its saved 'previous' snapshot is newer than the NameNode's current namespace state. Hadoop only permits rollback to a state that is older than or equal to the namespace: the previous layout version must be greater than or equal to (numerically, Hadoop layout versions count down as features are added) the current layout version, and the previous cTime must not exceed the namespace cTime. If either check fails, the DataNode refuses to roll back because rolling 'forward' the namespace state is not a supported operation.","triggerScenarios":"Running 'hdfs namenode -rollback' / starting the DataNode with rollback when the DataNode's previous/ directory was written by a newer Hadoop release (lower layout version number) or a later namespace creation time than the running NameNode's state. Concretely triggered when !(prevInfo.getLayoutVersion() >= DataNodeLayoutVersion.getCurrentLayoutVersion() && prevInfo.getCTime() <= nsInfo.getCTime()).","commonSituations":"NameNode was rolled back to an older image but some DataNodes had already upgraded past it; mixed Hadoop versions across the cluster after a partial upgrade/downgrade; NameNode re-formatted (fresh cTime) while DataNodes still hold upgraded state; a rollback attempted after the previous checkpoint was itself the product of a newer release.","solutions":["Roll the NameNode back first ('hdfs namenode -rollback') and verify its LV/CTime, then restart DataNodes with '-rollback', so the namespace is not older than the DataNode's previous state","Verify the whole cluster runs one Hadoop version (hdfs version on each node) so DataNode previous-state layout version is not newer than the NameNode's","If the NameNode state is fresh (reformatted or newer cTime needed), perform a full upgrade with '-upgrade' instead of a rollback","As a last resort on a throwaway cluster, clear the DataNode storage directory (or the previous/ and current/ dirs) and let the node re-register and re-replicate its blocks"],"exampleFix":"# before: NN left on older state, DN previous state is newer -> InconsistentFSStateException\nhdfs --daemon stop datanode\nhdfs --daemon start datanode -rollback\n\n# after: roll NameNode back first so namespace >= DN previous state, then DN\nhdfs --daemon stop namenode\nhdfs namenode -rollback\nhdfs --daemon start namenode\nhdfs --daemon start datanode -rollback","handlingStrategy":"validation","validationCode":"// Before rollback, compare DN previous-state LV/CTime with the NN state\n// (read from previous/<role>.md or VERSION files after upgrade):\nProperties prev = loadVersionFile(new File(dnDir, \"current/BP-*/previous/VERSION\"));\nint prevLV = Integer.parseInt(prev.getProperty(\"layoutVersion\"));\nlong prevCT = Long.parseLong(prev.getProperty(\"cTime\"));\nif (!(prevLV >= currentLayoutVersion /* NN's LV */\n      && prevCT <= nnCTime)) {\n  throw new IllegalStateException(\n      \"DN previous state is newer than NN namespace - rollback NN first or upgrade instead\");\n}\n// safe to: hdfs --daemon start datanode -rollback","typeGuard":null,"tryCatchPattern":"try {\n  storage.rollback(sd, nsInfo);\n} catch (InconsistentFSStateException e) {\n  // abort rollback for this volume; report LV/CTime pair from the message\n  // do NOT delete previous/ - operator must align NN/DN versions first\n  log.error(\"Rollback refused for {}: {}\", sd.getRoot(), e.getMessage());\n}","preventionTips":["Always roll back the NameNode before rolling back DataNodes, and use the same Hadoop release across the cluster","Never reformat the NameNode between an upgrade and a planned rollback - a fresh cTime invalidates every DN rollback","Automate upgrade/rollback with scripts that check 'hdfs version' and fsimage LV on every node before acting","Keep the previous/ directories untouched until the rollback window has safely passed"],"tags":["hdfs","datanode","rollback","upgrade","layout-version","ctime"],"backgroundTag":"storage-layout-version-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}