apache/hadoop · error · IOException

Timed out waiting for discardSegments() response

Error message

Timed out waiting for discardSegments() response

What it means

discardSegments(startTxId) fanned out to all JournalNodes did not get every response within timeoutMs (dfs.qjm.operations.timeout, default 60000 ms); waitFor threw TimeoutException and it is wrapped here. The call waits for all loggers, so a single unresponsive JN times it out.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:785

      throw new IOException("Timed out waiting for doFinalize() response");
    }
  }
  
  @Override
  public void discardSegments(long startTxId) throws IOException {
    QuorumCall<AsyncLogger, Void> call = loggers.discardSegments(startTxId);
    try {
      call.waitFor(loggers.size(), loggers.size(), 0,
          timeoutMs, "discardSegments");
      if (call.countExceptions() > 0) {
        call.rethrowException(
            "Could not perform discardSegments of one or more JournalNodes");
      }
    } catch (InterruptedException e) {
      throw new IOException(
          "Interrupted waiting for discardSegments() response");
    } catch (TimeoutException e) {
      throw new IOException(
          "Timed out waiting for discardSegments() response");
    }
  }
  
  @Override
  public long getJournalCTime() throws IOException {
    QuorumCall<AsyncLogger, Long> call = loggers.getJournalCTime();
    try {
      call.waitFor(loggers.size(), loggers.size(), 0,
          timeoutMs, "getJournalCTime");
      
      if (call.countExceptions() > 0) {
        call.rethrowException("Could not journal CTime for one "
            + "more JournalNodes");
      }
      
      // Either they all return the same thing or this call fails, so we can
      // just return the first result.

View on GitHub (pinned to 2add963021)

Solutions

1. Confirm all JournalNodes are running and reachable from the NN before retrying the rollback/discard step.
2. Check each JN's logs: those that did complete may already have discarded the segments — do not blindly re-run without checking dir state.
3. Raise dfs.qjm.operations.timeout for JNs with large numbers of segments or slow disks.
4. Fix the slow JN root cause (disk, GC, network) or restore a lost JN from a healthy JN's directory copy.

Example fix

// hdfs-site.xml
<property>
  <name>dfs.qjm.operations.timeout</name>
  <value>180000</value>
</property>

// Before discarding segments, confirm every JN is serving
for (URI jn : sharedEditsUris) {
  requireJnHealthy(jn);
}

static boolean isDiscardTimeout(IOException ioe) {
  return ioe.getCause() instanceof TimeoutException
      && ioe.getMessage().contains("discardSegments");
}

try {
  qjm.discardSegments(startTxId);
} catch (IOException ioe) {
  if (isDiscardTimeout(ioe)) {
    auditWhichJnsDiscarded(); // some may have deleted segments already
    retryOnceAllJnsHealthy();
  } else {
    throw ioe;
  }
}

When it happens

Trigger: Running the rollback step that discards edit segments from startTxId when one or more JNs are down, unreachable, or slow; deleting many segment files on a slow disk can exceed the timeout by itself.

Common situations: A JN is stopped or its host unreachable when rollback starts; JN disk with thousands of stale segment files is slow to delete; congested network; tight dfs.qjm.operations.timeout in the deployment.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Timed out waiting for lockSharedStorage() response
• Timed out waiting for doFinalize() response
• Interrupted waiting for discardSegments() response
• Timed out waiting for getJournalCTime() response
• Timed out waiting " + timeoutMs + "ms for a quorum of nodes

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5a18327bc89569ce. Report an issue: GitHub.