apache/hadoop · critical · IOException

Timed out waiting " + timeoutMs + "ms for a quorum of nodes

Error message

Timed out waiting " + timeoutMs + "ms for a quorum of nodes to respond.

What it means

A quorum of JournalNodes did not acknowledge the operation within its per-operation timeout (dfs.qjournal.write-txns.timeout.ms for edits, dfs.qjournal.start-segment.timeout.ms for segment starts, dfs.qjournal.finalize-segment.timeout.ms for finalizes). QuorumCall.waitFor threw TimeoutException and it surfaces as IOException; with fewer than majority successes the edit was never committed.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLoggerSet.java:138

   * @return a map of successful results
   * @throws QuorumException if a quorum doesn't respond with success
   * @throws IOException if the thread is interrupted or times out
   */
  <V> Map<AsyncLogger, V> waitForWriteQuorum(QuorumCall<AsyncLogger, V> q,
      int timeoutMs, String operationName) throws IOException {
    int majority = getMajoritySize();
    try {
      q.waitFor(
          loggers.size(), // either all respond 
          majority, // or we get a majority successes
          majority, // or we get a majority failures,
          timeoutMs, operationName);
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      throw new IOException("Interrupted waiting " + timeoutMs + "ms for a " +
          "quorum of nodes to respond.");
    } catch (TimeoutException e) {
      throw new IOException("Timed out waiting " + timeoutMs + "ms for a " +
          "quorum of nodes to respond.");
    }
    
    if (q.countSuccesses() < majority) {
      q.rethrowException("Got too many exceptions to achieve quorum size " +
          getMajorityString());
    }
    
    return q.getResults();
  }
  
  /**
   * @return the number of nodes which are required to obtain a quorum.
   */
  int getMajoritySize() {
    return loggers.size() / 2 + 1;
  }

View on GitHub (pinned to 2add963021)

Solutions

Verify every JournalNode process is up and its RPC port (default 8485) is reachable from the NameNode host.
Raise the specific per-operation timeout: dfs.qjournal.write-txns.timeout.ms, dfs.qjournal.start-segment.timeout.ms, dfs.qjournal.finalize-segment.timeout.ms.
Inspect JournalNode GC logs and edits-directory disk latency; move the edits dir off slow storage.
Keep a majority (2 of 3) of JournalNodes healthy at all times; an uncommitted edit is safe to retry only after the cause is fixed, since QJM recovery decides the committed state.

Example fix

<!-- before -->
<property><name>dfs.qjournal.write-txns.timeout.ms</name><value>20000</value></property>

<!-- after: accommodate slow fsync / large batches -->
<property><name>dfs.qjournal.write-txns.timeout.ms</name><value>60000</value></property>

Defensive patterns

Strategy: retry

Validate before calling

// Preflight before heavy edit bursts: verify a majority of JournalNode RPC ports answer
// (default JN RPC port 8485) from the NameNode host.
import java.net.*;

static boolean quorumReachable(List<InetSocketAddress> jns, int majority) {
  int ok = 0;
  for (InetSocketAddress a : jns) {
    try (Socket s = new Socket()) {
      s.connect(a, 2000);
      ok++;
    } catch (IOException ignored) { }
  }
  return ok >= majority;
}

Try / catch

try {
  loggers.waitForWriteQuorum(q, timeoutMs, operationName);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Timed out waiting")) {
    // not committed: verify JournalNode majority health, then reissue;
    // QJM recovery resolves any doubt about the failed attempt
    verifyJournalNodesReachable();
    retryWithBackoff();
    return;
  }
  throw e;
}

Prevention

Run an odd number (3 or 5) of JournalNodes and monitor majority liveness with alerts.
Size dfs.qjournal.*.timeout.ms against measured fsync latency plus GC headroom on the JNs.
Keep JournalNode edits directories on low-latency storage, separate from heavy DataNode volumes.
Watch JournalNode GC logs; a full pause on one JN plus one offline JN is the classic majority-loss path.

When it happens

Trigger: Enough JournalNodes are unreachable or slow (process down, network partition, GC pause, slow fsync) that a majority cannot ack within timeoutMs; or the per-op timeout is tuned below real commit latency for large edit batches.

Common situations: 3-JournalNode QJM with two JNs down; JournalNode GC pauses or stalled edits disks; cross-rack latency between NN and JNs; big transactions (many-block files, EC, snapshots) inflating batch write time.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/39d6386db59118e2. Report an issue: GitHub.