apache/hadoop · error · IOException

Timed out waiting for doUpgrade() response

Error message

Timed out waiting for doUpgrade() response

What it means

doUpgrade() waits for every JournalNode to complete its local storage upgrade within dfs.qjm.operations.timeout. Any JournalNode that does not answer in time fails the whole step with this timeout, aborting the upgrade.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:698

    } catch (TimeoutException e) {
      throw new IOException("Timed out waiting for doPreUpgrade() response");
    }
  }

  @Override
  public void doUpgrade(Storage storage) throws IOException {
    QuorumCall<AsyncLogger, Void> call = loggers.doUpgrade(storage);
    try {
      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,
          "doUpgrade");
      
      if (call.countExceptions() > 0) {
        call.rethrowException("Could not perform upgrade of one or more JournalNodes");
      }
    } catch (InterruptedException e) {
      throw new IOException("Interrupted waiting for doUpgrade() response");
    } catch (TimeoutException e) {
      throw new IOException("Timed out waiting for doUpgrade() response");
    }
  }
  
  @Override
  public void doFinalize() throws IOException {
    QuorumCall<AsyncLogger, Void> call = loggers.doFinalize();
    try {
      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,
          "doFinalize");
      
      if (call.countExceptions() > 0) {
        call.rethrowException("Could not finalize one or more JournalNodes");
      }
    } catch (InterruptedException e) {
      throw new IOException("Interrupted waiting for doFinalize() response");
    } catch (TimeoutException e) {
      throw new IOException("Timed out waiting for doFinalize() response");
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Ensure every JournalNode is up and reachable from the NameNode — the step requires all JNs.
2. Raise dfs.qjm.operations.timeout and rerun the upgrade (the step is resumable).
3. Check slow JournalNode logs and disk throughput; upgrades copy or rewrite edits directories.
4. After rerun, confirm all JournalNodes report the target layout version.

Example fix

<!-- raise the QJM operation timeout before upgrading -->
<property>
  <name>dfs.qjm.operations.timeout</name>
  <value>180000</value>
</property>

// Preflight before doUpgrade: ALL JournalNodes must be up (not just a majority)
for (InetSocketAddress jn : journalNodeAddresses) {
  try (Socket s = new Socket()) {
    s.connect(jn, 2000);
  } catch (IOException e) {
    throw new IllegalStateException(
        "JournalNode unreachable before upgrade: " + jn, e);
  }
}

try {
  qjm.doUpgrade(storage);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Timed out waiting for doUpgrade")) {
    // fix the non-answering JN, then rerun; layout upgrades are resumable
    reportUnreachableJournalNodes();
    scheduleUpgradeRetry();
    return;
  }
  throw e;
}

When it happens

Trigger: Upgrading the NameNode while a JournalNode is down, firewalled, or slow upgrading its edits directories (large segment copies, slow disks); dfs.qjm.operations.timeout below actual JN upgrade time.

Common situations: HA rolling upgrades with one unhealthy JournalNode; JN I/O stalls during layout-version migration; cross-site JNs with high latency.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Timed out waiting for doPreUpgrade() response
• Interrupted waiting for doPreUpgrade() response
• Interrupted waiting for doUpgrade() response
• Interrupted waiting for doFinalize() response
• Interrupted waiting " + timeoutMs + "ms for a quorum of node

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/827c34fac3aaa992. Report an issue: GitHub.