apache/hadoop · error · IOException

Timed out waiting for doPreUpgrade() response

Error message

Timed out waiting for doPreUpgrade() response

What it means

doPreUpgrade() waits for every JournalNode to finish the pre-upgrade step (segment finalize plus directory preparation) within dfs.qjm.operations.timeout; a single JournalNode that does not answer in time fails the step with this timeout, blocking the upgrade.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:681

  @VisibleForTesting
  AsyncLoggerSet getLoggerSetForTests() {
    return loggers;
  }

  @Override
  public void doPreUpgrade() throws IOException {
    QuorumCall<AsyncLogger, Void> call = loggers.doPreUpgrade();
    try {
      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,
          "doPreUpgrade");
      
      if (call.countExceptions() > 0) {
        call.rethrowException("Could not do pre-upgrade of one or more JournalNodes");
      }
    } catch (InterruptedException e) {
      throw new IOException("Interrupted waiting for doPreUpgrade() response");
    } catch (TimeoutException e) {
      throw new IOException("Timed out waiting for doPreUpgrade() response");
    }
  }

  @Override
  public void doUpgrade(Storage storage) throws IOException {
    QuorumCall<AsyncLogger, Void> call = loggers.doUpgrade(storage);
    try {
      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,
          "doUpgrade");
      
      if (call.countExceptions() > 0) {
        call.rethrowException("Could not perform upgrade of one or more JournalNodes");
      }
    } catch (InterruptedException e) {
      throw new IOException("Interrupted waiting for doUpgrade() response");
    } catch (TimeoutException e) {
      throw new IOException("Timed out waiting for doUpgrade() response");
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Bring every JournalNode up and reachable — this step requires ALL JNs, not a majority.
2. Raise dfs.qjm.operations.timeout and rerun the upgrade command (it is resumable).
3. Check the slow JournalNode's logs and edits disk (finalize and directory renames can stall on I/O).
4. Re-run the pre-upgrade step after fixing the JN; verify no JN is left mid-preparation.

Example fix

<!-- raise the QJM operation timeout before upgrading -->
<property>
  <name>dfs.qjm.operations.timeout</name>
  <value>120000</value>
</property>

// Preflight before upgrading: doPreUpgrade requires ALL JournalNodes reachable
static boolean allJournalNodesReachableForUpgrade(List<InetSocketAddress> jns) {
  for (InetSocketAddress a : jns) {
    try (Socket s = new Socket()) {
      s.connect(a, 2000);
    } catch (IOException e) {
      return false;
    }
  }
  return true;
}

try {
  qjm.doPreUpgrade();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Timed out waiting for doPreUpgrade")) {
    // identify the JN that never answered, fix it, rerun the resumable upgrade step
    reportUnreachableJournalNodes();
    scheduleUpgradeRetry();
    return;
  }
  throw e;
}

When it happens

Trigger: Running a NameNode rolling upgrade while any JournalNode is down, unreachable (RPC port 8485 blocked), or slow finalizing its edit segment; timeout too small for large edits directories.

Common situations: Rolling upgrades of QJM-backed HA clusters with an unhealthy JournalNode; slow JN disks during segment finalize; upgrade scripts racing JN startup.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Timed out waiting for doUpgrade() response
• Interrupted waiting for doPreUpgrade() response
• Interrupted waiting for doUpgrade() response
• Interrupted waiting for doFinalize() response
• Interrupted waiting " + timeoutMs + "ms for a quorum of node

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ad04e0ce2139e667. Report an issue: GitHub.