{"record":{"id":"39d6386db59118e2","repo":"apache/hadoop","slug":"timed-out-waiting-timeoutms-ms-for-a-quorum","errorCode":null,"errorMessage":"Timed out waiting \" + timeoutMs + \"ms for a quorum of nodes to respond.","messagePattern":"Timed out waiting \" \\+ timeoutMs \\+ \"ms for a quorum of nodes to respond\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLoggerSet.java","lineNumber":138,"sourceCode":"   * @return a map of successful results\n   * @throws QuorumException if a quorum doesn't respond with success\n   * @throws IOException if the thread is interrupted or times out\n   */\n  <V> Map<AsyncLogger, V> waitForWriteQuorum(QuorumCall<AsyncLogger, V> q,\n      int timeoutMs, String operationName) throws IOException {\n    int majority = getMajoritySize();\n    try {\n      q.waitFor(\n          loggers.size(), // either all respond \n          majority, // or we get a majority successes\n          majority, // or we get a majority failures,\n          timeoutMs, operationName);\n    } catch (InterruptedException e) {\n      Thread.currentThread().interrupt();\n      throw new IOException(\"Interrupted waiting \" + timeoutMs + \"ms for a \" +\n          \"quorum of nodes to respond.\");\n    } catch (TimeoutException e) {\n      throw new IOException(\"Timed out waiting \" + timeoutMs + \"ms for a \" +\n          \"quorum of nodes to respond.\");\n    }\n    \n    if (q.countSuccesses() < majority) {\n      q.rethrowException(\"Got too many exceptions to achieve quorum size \" +\n          getMajorityString());\n    }\n    \n    return q.getResults();\n  }\n  \n  /**\n   * @return the number of nodes which are required to obtain a quorum.\n   */\n  int getMajoritySize() {\n    return loggers.size() / 2 + 1;\n  }\n  ","sourceCodeStart":120,"sourceCodeEnd":156,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLoggerSet.java#L120-L156","documentation":"A quorum of JournalNodes did not acknowledge the operation within its per-operation timeout (dfs.qjournal.write-txns.timeout.ms for edits, dfs.qjournal.start-segment.timeout.ms for segment starts, dfs.qjournal.finalize-segment.timeout.ms for finalizes). QuorumCall.waitFor threw TimeoutException and it surfaces as IOException; with fewer than majority successes the edit was never committed.","triggerScenarios":"Enough JournalNodes are unreachable or slow (process down, network partition, GC pause, slow fsync) that a majority cannot ack within timeoutMs; or the per-op timeout is tuned below real commit latency for large edit batches.","commonSituations":"3-JournalNode QJM with two JNs down; JournalNode GC pauses or stalled edits disks; cross-rack latency between NN and JNs; big transactions (many-block files, EC, snapshots) inflating batch write time.","solutions":["Verify every JournalNode process is up and its RPC port (default 8485) is reachable from the NameNode host.","Raise the specific per-operation timeout: dfs.qjournal.write-txns.timeout.ms, dfs.qjournal.start-segment.timeout.ms, dfs.qjournal.finalize-segment.timeout.ms.","Inspect JournalNode GC logs and edits-directory disk latency; move the edits dir off slow storage.","Keep a majority (2 of 3) of JournalNodes healthy at all times; an uncommitted edit is safe to retry only after the cause is fixed, since QJM recovery decides the committed state."],"exampleFix":"<!-- before -->\n<property><name>dfs.qjournal.write-txns.timeout.ms</name><value>20000</value></property>\n\n<!-- after: accommodate slow fsync / large batches -->\n<property><name>dfs.qjournal.write-txns.timeout.ms</name><value>60000</value></property>","handlingStrategy":"retry","validationCode":"// Preflight before heavy edit bursts: verify a majority of JournalNode RPC ports answer\n// (default JN RPC port 8485) from the NameNode host.\nimport java.net.*;\n\nstatic boolean quorumReachable(List<InetSocketAddress> jns, int majority) {\n  int ok = 0;\n  for (InetSocketAddress a : jns) {\n    try (Socket s = new Socket()) {\n      s.connect(a, 2000);\n      ok++;\n    } catch (IOException ignored) { }\n  }\n  return ok >= majority;\n}","typeGuard":null,"tryCatchPattern":"try {\n  loggers.waitForWriteQuorum(q, timeoutMs, operationName);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Timed out waiting\")) {\n    // not committed: verify JournalNode majority health, then reissue;\n    // QJM recovery resolves any doubt about the failed attempt\n    verifyJournalNodesReachable();\n    retryWithBackoff();\n    return;\n  }\n  throw e;\n}","preventionTips":["Run an odd number (3 or 5) of JournalNodes and monitor majority liveness with alerts.","Size dfs.qjournal.*.timeout.ms against measured fsync latency plus GC headroom on the JNs.","Keep JournalNode edits directories on low-latency storage, separate from heavy DataNode volumes.","Watch JournalNode GC logs; a full pause on one JN plus one offline JN is the classic majority-loss path."],"tags":["hdfs","qjm","journal-node","timeout","quorum","edit-log"],"backgroundTag":"quorum-timeout","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}