apache/hadoop · error · IOException

Interrupted waiting " + timeoutMs + "ms for a quorum of node

Error message

Interrupted waiting " + timeoutMs + "ms for a quorum of nodes to respond.

What it means

AsyncLoggerSet.waitForWriteQuorum() is how QuorumJournalManager commits edits: it waits until all JournalNodes answer or a majority succeed/fail. InterruptedException means the waiting thread (normally the NameNode's edit-log writer) was interrupted mid-wait; the handler restores the interrupt flag and converts it to IOException.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLoggerSet.java:135

   * @param q the quorum call
   * @param timeoutMs the number of millis to wait
   * @param operationName textual description of the operation, for logging
   * @return a map of successful results
   * @throws QuorumException if a quorum doesn't respond with success
   * @throws IOException if the thread is interrupted or times out
   */
  <V> Map<AsyncLogger, V> waitForWriteQuorum(QuorumCall<AsyncLogger, V> q,
      int timeoutMs, String operationName) throws IOException {
    int majority = getMajoritySize();
    try {
      q.waitFor(
          loggers.size(), // either all respond 
          majority, // or we get a majority successes
          majority, // or we get a majority failures,
          timeoutMs, operationName);
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      throw new IOException("Interrupted waiting " + timeoutMs + "ms for a " +
          "quorum of nodes to respond.");
    } catch (TimeoutException e) {
      throw new IOException("Timed out waiting " + timeoutMs + "ms for a " +
          "quorum of nodes to respond.");
    }
    
    if (q.countSuccesses() < majority) {
      q.rethrowException("Got too many exceptions to achieve quorum size " +
          getMajorityString());
    }
    
    return q.getResults();
  }
  
  /**
   * @return the number of nodes which are required to obtain a quorum.
   */
  int getMajoritySize() {

View on GitHub (pinned to 2add963021)

Solutions

Correlate the timestamp with NN shutdown or failover events; if they match, no action is needed.
If it happens spontaneously, take a jstack of the NameNode to find which thread interrupts the edit-log writer.
After failover, confirm the new active performed segment recovery (automatic via QJM epoch recovery).

Defensive patterns

Strategy: try-catch

Try / catch

try {
  Map<AsyncLogger, V> results = loggers.waitForWriteQuorum(call, timeoutMs, op);
} catch (IOException e) {
  if (Thread.currentThread().isInterrupted()
      || (e.getMessage() != null && e.getMessage().startsWith("Interrupted waiting"))) {
    Thread.currentThread().interrupt();
    // cancellation (shutdown/failover): stop cleanly, do not retry the write
    throw new CancellationException("quorum wait interrupted: " + e.getMessage());
  }
  throw e; // real I/O failure: handle separately
}

Prevention

Never interrupt the NameNode edit-log writer except for shutdown/failover; document that contract in tooling.
Correlate this exception with shutdown/failover events before debugging it as a fault.
Keep HA failover thresholds above expected quorum-write latency so writers are not cancelled mid-commit.

When it happens

Trigger: The FSEditLog writer thread is interrupted while a quorum write is in flight — typically NameNode shutdown, HA failover aborting the writer, or tooling cancelling the thread.

Common situations: Active NN shutdown or ZKFC-initiated failover during heavy edit traffic; restart of the active during metadata operations; expected and benign when it lines up with an intentional stop/failover.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/70361ba8b2c12cb5. Report an issue: GitHub.