{"record":{"id":"ad04e0ce2139e667","repo":"apache/hadoop","slug":"timed-out-waiting-for-dopreupgrade-response","errorCode":null,"errorMessage":"Timed out waiting for doPreUpgrade() response","messagePattern":"Timed out waiting for doPreUpgrade\\(\\) response","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java","lineNumber":681,"sourceCode":"  @VisibleForTesting\n  AsyncLoggerSet getLoggerSetForTests() {\n    return loggers;\n  }\n\n  @Override\n  public void doPreUpgrade() throws IOException {\n    QuorumCall<AsyncLogger, Void> call = loggers.doPreUpgrade();\n    try {\n      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,\n          \"doPreUpgrade\");\n      \n      if (call.countExceptions() > 0) {\n        call.rethrowException(\"Could not do pre-upgrade of one or more JournalNodes\");\n      }\n    } catch (InterruptedException e) {\n      throw new IOException(\"Interrupted waiting for doPreUpgrade() response\");\n    } catch (TimeoutException e) {\n      throw new IOException(\"Timed out waiting for doPreUpgrade() response\");\n    }\n  }\n\n  @Override\n  public void doUpgrade(Storage storage) throws IOException {\n    QuorumCall<AsyncLogger, Void> call = loggers.doUpgrade(storage);\n    try {\n      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,\n          \"doUpgrade\");\n      \n      if (call.countExceptions() > 0) {\n        call.rethrowException(\"Could not perform upgrade of one or more JournalNodes\");\n      }\n    } catch (InterruptedException e) {\n      throw new IOException(\"Interrupted waiting for doUpgrade() response\");\n    } catch (TimeoutException e) {\n      throw new IOException(\"Timed out waiting for doUpgrade() response\");\n    }","sourceCodeStart":663,"sourceCodeEnd":699,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java#L663-L699","documentation":"doPreUpgrade() waits for every JournalNode to finish the pre-upgrade step (segment finalize plus directory preparation) within dfs.qjm.operations.timeout; a single JournalNode that does not answer in time fails the step with this timeout, blocking the upgrade.","triggerScenarios":"Running a NameNode rolling upgrade while any JournalNode is down, unreachable (RPC port 8485 blocked), or slow finalizing its edit segment; timeout too small for large edits directories.","commonSituations":"Rolling upgrades of QJM-backed HA clusters with an unhealthy JournalNode; slow JN disks during segment finalize; upgrade scripts racing JN startup.","solutions":["Bring every JournalNode up and reachable — this step requires ALL JNs, not a majority.","Raise dfs.qjm.operations.timeout and rerun the upgrade command (it is resumable).","Check the slow JournalNode's logs and edits disk (finalize and directory renames can stall on I/O).","Re-run the pre-upgrade step after fixing the JN; verify no JN is left mid-preparation."],"exampleFix":"<!-- raise the QJM operation timeout before upgrading -->\n<property>\n  <name>dfs.qjm.operations.timeout</name>\n  <value>120000</value>\n</property>","handlingStrategy":"retry","validationCode":"// Preflight before upgrading: doPreUpgrade requires ALL JournalNodes reachable\nstatic boolean allJournalNodesReachableForUpgrade(List<InetSocketAddress> jns) {\n  for (InetSocketAddress a : jns) {\n    try (Socket s = new Socket()) {\n      s.connect(a, 2000);\n    } catch (IOException e) {\n      return false;\n    }\n  }\n  return true;\n}","typeGuard":null,"tryCatchPattern":"try {\n  qjm.doPreUpgrade();\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Timed out waiting for doPreUpgrade\")) {\n    // identify the JN that never answered, fix it, rerun the resumable upgrade step\n    reportUnreachableJournalNodes();\n    scheduleUpgradeRetry();\n    return;\n  }\n  throw e;\n}","preventionTips":["Health-check every JournalNode before starting a rolling upgrade; the step needs all of them.","Raise dfs.qjm.operations.timeout for the upgrade window, then restore it.","Watch JN disk throughput during upgrades — segment finalize is I/O-heavy."],"tags":["hdfs","qjm","journal-node","upgrade","timeout","ha"],"backgroundTag":"quorum-timeout","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}