{"record":{"id":"5a18327bc89569ce","repo":"apache/hadoop","slug":"timed-out-waiting-for-discardsegments-response","errorCode":null,"errorMessage":"Timed out waiting for discardSegments() response","messagePattern":"Timed out waiting for discardSegments\\(\\) response","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java","lineNumber":785,"sourceCode":"      throw new IOException(\"Timed out waiting for doFinalize() response\");\n    }\n  }\n  \n  @Override\n  public void discardSegments(long startTxId) throws IOException {\n    QuorumCall<AsyncLogger, Void> call = loggers.discardSegments(startTxId);\n    try {\n      call.waitFor(loggers.size(), loggers.size(), 0,\n          timeoutMs, \"discardSegments\");\n      if (call.countExceptions() > 0) {\n        call.rethrowException(\n            \"Could not perform discardSegments of one or more JournalNodes\");\n      }\n    } catch (InterruptedException e) {\n      throw new IOException(\n          \"Interrupted waiting for discardSegments() response\");\n    } catch (TimeoutException e) {\n      throw new IOException(\n          \"Timed out waiting for discardSegments() response\");\n    }\n  }\n  \n  @Override\n  public long getJournalCTime() throws IOException {\n    QuorumCall<AsyncLogger, Long> call = loggers.getJournalCTime();\n    try {\n      call.waitFor(loggers.size(), loggers.size(), 0,\n          timeoutMs, \"getJournalCTime\");\n      \n      if (call.countExceptions() > 0) {\n        call.rethrowException(\"Could not journal CTime for one \"\n            + \"more JournalNodes\");\n      }\n      \n      // Either they all return the same thing or this call fails, so we can\n      // just return the first result.","sourceCodeStart":767,"sourceCodeEnd":803,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java#L767-L803","documentation":"discardSegments(startTxId) fanned out to all JournalNodes did not get every response within timeoutMs (dfs.qjm.operations.timeout, default 60000 ms); waitFor threw TimeoutException and it is wrapped here. The call waits for all loggers, so a single unresponsive JN times it out.","triggerScenarios":"Running the rollback step that discards edit segments from startTxId when one or more JNs are down, unreachable, or slow; deleting many segment files on a slow disk can exceed the timeout by itself.","commonSituations":"A JN is stopped or its host unreachable when rollback starts; JN disk with thousands of stale segment files is slow to delete; congested network; tight dfs.qjm.operations.timeout in the deployment.","solutions":["Confirm all JournalNodes are running and reachable from the NN before retrying the rollback/discard step.","Check each JN's logs: those that did complete may already have discarded the segments — do not blindly re-run without checking dir state.","Raise dfs.qjm.operations.timeout for JNs with large numbers of segments or slow disks.","Fix the slow JN root cause (disk, GC, network) or restore a lost JN from a healthy JN's directory copy."],"exampleFix":"// hdfs-site.xml\n<property>\n  <name>dfs.qjm.operations.timeout</name>\n  <value>180000</value>\n</property>","handlingStrategy":"retry","validationCode":"// Before discarding segments, confirm every JN is serving\nfor (URI jn : sharedEditsUris) {\n  requireJnHealthy(jn);\n}","typeGuard":"static boolean isDiscardTimeout(IOException ioe) {\n  return ioe.getCause() instanceof TimeoutException\n      && ioe.getMessage().contains(\"discardSegments\");\n}","tryCatchPattern":"try {\n  qjm.discardSegments(startTxId);\n} catch (IOException ioe) {\n  if (isDiscardTimeout(ioe)) {\n    auditWhichJnsDiscarded(); // some may have deleted segments already\n    retryOnceAllJnsHealthy();\n  } else {\n    throw ioe;\n  }\n}","preventionTips":["Verify the full JN set is up before starting the rollback phase that discards segments.","Keep JN disks uncluttered — huge numbers of stale segments make deletion exceed the timeout.","Raise dfs.qjm.operations.timeout for slow-storage JNs."],"tags":["qjm","rollback","discard-segments","timeout","journalnode"],"backgroundTag":"rpc-quorum-timeout","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}