{"record":{"id":"5933d0bfa4bfc8a7","repo":"apache/hadoop","slug":"results-differed-for-getjournalctime","errorCode":null,"errorMessage":"Results differed for getJournalCTime","messagePattern":"Results differed for getJournalCTime","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java","lineNumber":807,"sourceCode":"  \n  @Override\n  public long getJournalCTime() throws IOException {\n    QuorumCall<AsyncLogger, Long> call = loggers.getJournalCTime();\n    try {\n      call.waitFor(loggers.size(), loggers.size(), 0,\n          timeoutMs, \"getJournalCTime\");\n      \n      if (call.countExceptions() > 0) {\n        call.rethrowException(\"Could not journal CTime for one \"\n            + \"more JournalNodes\");\n      }\n      \n      // Either they all return the same thing or this call fails, so we can\n      // just return the first result.\n      try {\n        DFSUtil.assertAllResultsEqual(call.getResults().values());\n      } catch (AssertionError ae) {\n        throw new IOException(\"Results differed for getJournalCTime\", ae);\n      }\n      for (Long result : call.getResults().values()) {\n        return result;\n      }\n    } catch (InterruptedException e) {\n      throw new IOException(\"Interrupted waiting for getJournalCTime() \" +\n          \"response\");\n    } catch (TimeoutException e) {\n      throw new IOException(\"Timed out waiting for getJournalCTime() \" +\n          \"response\");\n    }\n    \n    throw new AssertionError(\"Unreachable code.\");\n  }\n}\n","sourceCodeStart":789,"sourceCodeEnd":823,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java#L789-L823","documentation":"getJournalCTime() asks every JournalNode for its journal creation/upgrade time and requires all results to be equal (DFSUtil.assertAllResultsEqual). Divergent cTime values across JNs — meaning the nodes were formatted or upgraded at different moments — produce this IOException. The NN uses this during upgrade startup to sanity-check shared edits consistency.","triggerScenarios":"Calling getJournalCTime() during upgrade-related startup when JN dirs were created/upgraded at different times: one JN initialized later, one JN restored from backup, one JN that missed a completed upgrade, or a JN reformatted by hand.","commonSituations":"Adding a replacement JournalNode that was freshly initialized while others kept original state; restoring one JN from an old backup; a JN that was down during an 'hdfs namenode -upgrade' and never caught up; mixing dirs from a different cluster.","solutions":["Compare the cTime field in each JN's current dir VERSION file (dfs.journalnode.edits.dir/<jid>/current/VERSION) to identify the divergent node.","If the majority share one cTime, overwrite the odd JN's journal dir with a copy of a healthy JN's dir (stop the JN first, copy, restart).","Alternatively rebuild shared edits wholesale: stop NNs, correct NN storage, run 'hdfs namenode -initializeSharedEdits' to lay down identical state on all JNs.","Verify JN address configuration — a JN from another cluster in dfs.namenode.shared.edits.dir also shows up as a cTime mismatch."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Preflight: fetch cTime from each JN and require unanimity\nList<Long> ctimes = queryEachJnCtime(sharedEditsUris);\nif (ctimes.stream().distinct().count() > 1) {\n  throw new IllegalStateException(\"JN cTimes diverge: \" + ctimes);\n}","typeGuard":"static boolean isCTimeMismatch(IOException ioe) {\n  return ioe.getMessage() != null\n      && ioe.getMessage().contains(\"Results differed for getJournalCTime\");\n}","tryCatchPattern":"try {\n  long ct = qjm.getJournalCTime();\n} catch (IOException ioe) {\n  if (isCTimeMismatch(ioe)) {\n    // data problem, not transient: stop the upgrade and reconcile JN dirs\n    haltAndAlertAdmin(ioe);\n  } else {\n    throw ioe;\n  }\n}","preventionTips":["Initialize all JournalNodes at cluster creation time from a single 'hdfs namenode -initializeSharedEdits' run so cTime starts identical.","When replacing a JN, copy its dir from a healthy peer instead of formatting fresh.","Diff cTime across JN VERSION files as part of upgrade pre-checks."],"tags":["qjm","upgrade","ctime","state-divergence","consensus-check"],"backgroundTag":"quorum-result-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}