{"record":{"id":"20a8b55008fc40c1","repo":"apache/hadoop","slug":"timed-out-waiting-for-getjournalctime-response","errorCode":null,"errorMessage":"Timed out waiting for getJournalCTime() response","messagePattern":"Timed out waiting for getJournalCTime\\(\\) response","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java","lineNumber":816,"sourceCode":"        call.rethrowException(\"Could not journal CTime for one \"\n            + \"more JournalNodes\");\n      }\n      \n      // Either they all return the same thing or this call fails, so we can\n      // just return the first result.\n      try {\n        DFSUtil.assertAllResultsEqual(call.getResults().values());\n      } catch (AssertionError ae) {\n        throw new IOException(\"Results differed for getJournalCTime\", ae);\n      }\n      for (Long result : call.getResults().values()) {\n        return result;\n      }\n    } catch (InterruptedException e) {\n      throw new IOException(\"Interrupted waiting for getJournalCTime() \" +\n          \"response\");\n    } catch (TimeoutException e) {\n      throw new IOException(\"Timed out waiting for getJournalCTime() \" +\n          \"response\");\n    }\n    \n    throw new AssertionError(\"Unreachable code.\");\n  }\n}\n","sourceCodeStart":798,"sourceCodeEnd":823,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java#L798-L823","documentation":"getJournalCTime() waited for every JournalNode to report its cTime and the full set of responses did not arrive within timeoutMs (dfs.qjm.operations.timeout, default 60000 ms); the TimeoutException is wrapped in this IOException. As elsewhere in these all-JN waits, one missing answer times out the call.","triggerScenarios":"The upgrade-startup cTime check runs while a JournalNode is down, unreachable, or slow; any JN that fails to answer within dfs.qjm.operations.timeout triggers the timeout.","commonSituations":"A JN host is down when the NN starts with '-upgrade'; a JN's disk is slow to read its VERSION/segment metadata; network problems between NN and one JN; default 60s too tight for slow storage.","solutions":["Start all JournalNodes and confirm their RPC ports answer from the NN host, then retry the NN startup/upgrade.","Investigate the non-answering JN's logs for disk or GC stalls and fix them.","Increase dfs.qjm.operations.timeout if the JNs are healthy but occasionally slow.","Restore a permanently lost JN from another JN's journal directory before retrying."],"exampleFix":"// hdfs-site.xml\n<property>\n  <name>dfs.qjm.operations.timeout</name>\n  <value>120000</value>\n</property>","handlingStrategy":"retry","validationCode":"// Confirm each JN answers before the NN runs its cTime check\nrequireAllJnsReachable(sharedEditsUris);","typeGuard":"static boolean isCTimeTimeout(IOException ioe) {\n  return ioe.getCause() instanceof TimeoutException\n      && ioe.getMessage().contains(\"getJournalCTime\");\n}","tryCatchPattern":"try {\n  long ct = qjm.getJournalCTime();\n} catch (IOException ioe) {\n  if (isCTimeTimeout(ioe)) {\n    retryAfterJnHealthCheck();\n  } else {\n    throw ioe;\n  }\n}","preventionTips":["All JNs up before '-upgrade' startup — the check is all-JN, not quorum.","Alert on JN response latency; a JN nearing dfs.qjm.operations.timeout will break this check.","Tune dfs.qjm.operations.timeout for the environment."],"tags":["qjm","upgrade","ctime","timeout","journalnode"],"backgroundTag":"rpc-quorum-timeout","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}