{"record":{"id":"827c34fac3aaa992","repo":"apache/hadoop","slug":"timed-out-waiting-for-doupgrade-response","errorCode":null,"errorMessage":"Timed out waiting for doUpgrade() response","messagePattern":"Timed out waiting for doUpgrade\\(\\) response","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java","lineNumber":698,"sourceCode":"    } catch (TimeoutException e) {\n      throw new IOException(\"Timed out waiting for doPreUpgrade() response\");\n    }\n  }\n\n  @Override\n  public void doUpgrade(Storage storage) throws IOException {\n    QuorumCall<AsyncLogger, Void> call = loggers.doUpgrade(storage);\n    try {\n      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,\n          \"doUpgrade\");\n      \n      if (call.countExceptions() > 0) {\n        call.rethrowException(\"Could not perform upgrade of one or more JournalNodes\");\n      }\n    } catch (InterruptedException e) {\n      throw new IOException(\"Interrupted waiting for doUpgrade() response\");\n    } catch (TimeoutException e) {\n      throw new IOException(\"Timed out waiting for doUpgrade() response\");\n    }\n  }\n  \n  @Override\n  public void doFinalize() throws IOException {\n    QuorumCall<AsyncLogger, Void> call = loggers.doFinalize();\n    try {\n      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,\n          \"doFinalize\");\n      \n      if (call.countExceptions() > 0) {\n        call.rethrowException(\"Could not finalize one or more JournalNodes\");\n      }\n    } catch (InterruptedException e) {\n      throw new IOException(\"Interrupted waiting for doFinalize() response\");\n    } catch (TimeoutException e) {\n      throw new IOException(\"Timed out waiting for doFinalize() response\");\n    }","sourceCodeStart":680,"sourceCodeEnd":716,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java#L680-L716","documentation":"doUpgrade() waits for every JournalNode to complete its local storage upgrade within dfs.qjm.operations.timeout. Any JournalNode that does not answer in time fails the whole step with this timeout, aborting the upgrade.","triggerScenarios":"Upgrading the NameNode while a JournalNode is down, firewalled, or slow upgrading its edits directories (large segment copies, slow disks); dfs.qjm.operations.timeout below actual JN upgrade time.","commonSituations":"HA rolling upgrades with one unhealthy JournalNode; JN I/O stalls during layout-version migration; cross-site JNs with high latency.","solutions":["Ensure every JournalNode is up and reachable from the NameNode — the step requires all JNs.","Raise dfs.qjm.operations.timeout and rerun the upgrade (the step is resumable).","Check slow JournalNode logs and disk throughput; upgrades copy or rewrite edits directories.","After rerun, confirm all JournalNodes report the target layout version."],"exampleFix":"<!-- raise the QJM operation timeout before upgrading -->\n<property>\n  <name>dfs.qjm.operations.timeout</name>\n  <value>180000</value>\n</property>","handlingStrategy":"retry","validationCode":"// Preflight before doUpgrade: ALL JournalNodes must be up (not just a majority)\nfor (InetSocketAddress jn : journalNodeAddresses) {\n  try (Socket s = new Socket()) {\n    s.connect(jn, 2000);\n  } catch (IOException e) {\n    throw new IllegalStateException(\n        \"JournalNode unreachable before upgrade: \" + jn, e);\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  qjm.doUpgrade(storage);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Timed out waiting for doUpgrade\")) {\n    // fix the non-answering JN, then rerun; layout upgrades are resumable\n    reportUnreachableJournalNodes();\n    scheduleUpgradeRetry();\n    return;\n  }\n  throw e;\n}","preventionTips":["Never start a rolling upgrade with a degraded JournalNode set; upgrade steps wait for ALL JNs.","Budget dfs.qjm.operations.timeout for the largest edits directory in the JN set.","Confirm all JournalNodes report the target layout version after a successful rerun."],"tags":["hdfs","qjm","journal-node","upgrade","timeout","ha"],"backgroundTag":"quorum-timeout","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}