{"record":{"id":"4edbe40ceafb5594","repo":"apache/hadoop","slug":"datanode-ctime-is-not-equal-to-namenode-ctim","errorCode":null,"errorMessage":"Datanode CTime (={}) is not equal to namenode CTime (={})","messagePattern":"Datanode CTime \\(=(.+?)\\) is not equal to namenode CTime \\(=(.+?)\\)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java","lineNumber":185,"sourceCode":"      case NOT_FORMATTED: // format\n        LOG.info(\"Block pool storage directory for location {} and block pool\"\n                + \" id {} is not formatted. Formatting ...\", location,\n            nsInfo.getBlockPoolID());\n        format(sd, nsInfo);\n        break;\n      default:  // recovery part is common\n        sd.doRecover(curState);\n      }\n\n      // 2. Do transitions\n      // Each storage directory is treated individually.\n      // During startup some of them can upgrade or roll back\n      // while others could be up-to-date for the regular startup.\n      if (!doTransition(sd, nsInfo, startOpt, callables, conf)) {\n\n        // 3. Check CTime and update successfully loaded storage.\n        if (getCTime() != nsInfo.getCTime()) {\n          throw new IOException(\"Datanode CTime (=\" + getCTime()\n              + \") is not equal to namenode CTime (=\" + nsInfo.getCTime() + \")\");\n        }\n        setServiceLayoutVersion(getServiceLayoutVersion());\n        writeProperties(sd);\n      }\n\n      return sd;\n    } catch (IOException ioe) {\n      sd.unlock();\n      throw ioe;\n    }\n  }\n\n  /**\n   * Analyze and load storage directories. Recover from previous transitions if\n   * required.\n   *\n   * The block pool storages are either all analyzed or none of them is loaded.","sourceCodeStart":167,"sourceCodeEnd":203,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java#L167-L203","documentation":"After doTransition reported that no layout transition was needed, the DN found its persisted block pool cTime differs from the NameNode's cTime. DN storage state and namespace state disagree, which in practice means an upgrade/rollback of the namespace happened without the matching DN-side transition.","triggerScenarios":"NN was upgraded or its cTime otherwise bumped while the DN starts with plain -regular; NN re-formatted; an aborted upgrade left BP storage at the old cTime with no transition path selected.","commonSituations":"Operators re-format the NN and restart DNs normally; rolling upgrade finalized inconsistently; test clusters reused after NN reset.","solutions":["Restart the DN with -upgrade (or -rollback, matching what the namespace went through)","If the NN was deliberately re-formatted, delete the DN's stale block pool dirs so they re-register fresh","Restore the NN namespace from backup if the cTime change was accidental"],"exampleFix":"# before\nhdfs --daemon start datanode            # -regular, throws\n\n# after (namespace was upgraded)\nhdfs --daemon start datanode -upgrade\n# or, if NN rolled back:\nhdfs --daemon start datanode -rollback","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (IOException e) {\n  if (e.getMessage().contains(\"is not equal to namenode CTime\")) {\n    // restart DN with -upgrade or -rollback matching the namespace's transition\n    failFastWithGuidance(e);\n  }\n}","preventionTips":["Always restart DNs with the startup option matching what the namespace went through (-upgrade/-rollback)","Never re-format the NameNode without a plan for DN storage cleanup","Monitor DN logs for cTime complaints after every NN state change"],"tags":["hdfs","datanode","storage","ctime","upgrade-mismatch"],"backgroundTag":"storage-ctime-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}