apache/hadoop · error · IOException

Timed out waiting for doFinalize() response

Error message

Timed out waiting for doFinalize() response

What it means

QuorumJournalManager.doFinalize() fans out a doFinalize RPC to every JournalNode listed in dfs.namenode.shared.edits.dir and blocks until ALL of them answer (waitFor is called with min=max=loggers.size()). If the full set has not responded within timeoutMs (dfs.qjm.operations.timeout, default 60000ms), waitFor throws TimeoutException, which is wrapped in this IOException. It is thrown during NameNode upgrade finalization that touches the shared edits (QJM) storage.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:715

    } catch (TimeoutException e) {
      throw new IOException("Timed out waiting for doUpgrade() response");
    }
  }
  
  @Override
  public void doFinalize() throws IOException {
    QuorumCall<AsyncLogger, Void> call = loggers.doFinalize();
    try {
      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,
          "doFinalize");
      
      if (call.countExceptions() > 0) {
        call.rethrowException("Could not finalize one or more JournalNodes");
      }
    } catch (InterruptedException e) {
      throw new IOException("Interrupted waiting for doFinalize() response");
    } catch (TimeoutException e) {
      throw new IOException("Timed out waiting for doFinalize() response");
    }
  }
  
  @Override
  public boolean canRollBack(StorageInfo storage, StorageInfo prevStorage,
      int targetLayoutVersion) throws IOException {
    QuorumCall<AsyncLogger, Boolean> call = loggers.canRollBack(storage,
        prevStorage, targetLayoutVersion);
    try {
      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,
          "lockSharedStorage");
      
      if (call.countExceptions() > 0) {
        call.rethrowException("Could not check if roll back possible for"
            + " one or more JournalNodes");
      }
      
      // Either they all return the same thing or this call fails, so we can

View on GitHub (pinned to 2add963021)

Solutions

1. Verify every JournalNode in dfs.namenode.shared.edits.dir is running and its RPC port is reachable from the NameNode host, then retry the finalize operation.
2. Inspect the slow/unreachable JN's logs for disk errors (full disk, bad disk, slow fsync) or GC pauses and fix the underlying slowness.
3. Raise dfs.qjm.operations.timeout above the slowest observed JN finalize time and retry (it defaults to 60000 ms).
4. If a JN is permanently lost, restore its journal directory from an identical copy of a healthy JN's directory (or fix the host) before retrying — doFinalize requires answers from all JNs, not a quorum.

Example fix

// before
<property>
  <name>dfs.qjm.operations.timeout</name>
  <value>20000</value>
</property>

<!-- after: give slow JournalNodes time to answer doFinalize -->
<property>
  <name>dfs.qjm.operations.timeout</name>
  <value>120000</value>
</property>

// Preflight before doFinalize: every JN HTTP endpoint must answer
for (URI jn : sharedEditsUris) {
  URL jstatus = new URL("http", jn.getHost(), 8480, "/jstatus");
  try (InputStream in = jstatus.openStream()) {
    // JournalNode is alive and serving
  } catch (IOException e) {
    throw new IllegalStateException("JN not reachable before finalize: " + jn, e);
  }
}

static boolean isFinalizeTimeout(IOException ioe) {
  return ioe.getCause() instanceof TimeoutException
      && ioe.getMessage().contains("doFinalize");
}

try {
  qjm.doFinalize();
} catch (IOException ioe) {
  if (ioe.getCause() instanceof TimeoutException) {
    // transient: retry after confirming JN health, with backoff
  } else {
    throw ioe; // real per-JN failure, not a timeout
  }
}

When it happens

Trigger: Invoking the QJM doFinalize() path (JournalManager upgrade-finalize, e.g. 'hdfs namenode -finalize'-class operations / upgrade finalization with shared edits) while at least one JournalNode is down, unreachable, or slower than dfs.qjm.operations.timeout; because the call waits for every logger, a single unresponsive JN triggers the timeout.

Common situations: One of the three JournalNodes is stopped or its host is down during finalize; a JN disk is failing or full so the local finalize RPC stalls; long GC pause on a JN; network partition between NN and one JN; retrying a finalize after a previous partial run; test/CI environments with a tight dfs.qjm.operations.timeout.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Timed out waiting for lockSharedStorage() response
• Timed out waiting for discardSegments() response
• Timed out waiting for getJournalCTime() response
• Timed out waiting " + timeoutMs + "ms for a quorum of nodes
• Timed out waiting for format() response

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6ad425fb40416e5c. Report an issue: GitHub.