apache/hadoop · error · IOException

Timed out waiting for format() response

Error message

Timed out waiting for format() response

What it means

format() requires every JournalNode to acknowledge (QuorumCall.waitFor with loggers.size() successes) within dfs.qjm.operations.timeout. A single unreachable or slow JournalNode therefore fails the whole format with this timeout.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:260

    long myEpoch = maxPromised + 1;
    Map<AsyncLogger, NewEpochResponseProto> resps =
        loggers.waitForWriteQuorum(loggers.newEpoch(nsInfo, myEpoch),
            newEpochTimeoutMs, "newEpoch(" + myEpoch + ")");
        
    loggers.setEpoch(myEpoch);
    return resps;
  }
  
  @Override
  public void format(NamespaceInfo nsInfo, boolean force) throws IOException {
    QuorumCall<AsyncLogger, Void> call = loggers.format(nsInfo, force);
    try {
      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,
          "format");
    } catch (InterruptedException e) {
      throw new IOException("Interrupted waiting for format() response");
    } catch (TimeoutException e) {
      throw new IOException("Timed out waiting for format() response");
    }
    
    if (call.countExceptions() > 0) {
      call.rethrowException("Could not format one or more JournalNodes");
    }
  }

  @Override
  public boolean hasSomeData() throws IOException {
    QuorumCall<AsyncLogger, Boolean> call =
        loggers.isFormatted();

    try {
      call.waitFor(loggers.size(), 0, 0, timeoutMs, "hasSomeData");
    } catch (InterruptedException e) {
      throw new IOException("Interrupted while determining if JNs have data");
    } catch (TimeoutException e) {
      throw new IOException("Timed out waiting for response from loggers");

View on GitHub (pinned to 2add963021)

Solutions

1. Start every JournalNode and verify from the NameNode host that each JN RPC port answers.
2. Raise dfs.qjm.operations.timeout if JournalNodes are slow, then rerun the format.
3. Check JournalNode logs for errors handling the format RPC (locked edits directories, permissions).
4. Once the full JN set is healthy, rerun format (with -force) because a partially formatted quorum must be redone.

Example fix

<!-- raise the QJM operation timeout before formatting -->
<property>
  <name>dfs.qjm.operations.timeout</name>
  <value>120000</value>
</property>

// Preflight: every JournalNode must answer before 'hdfs namenode -format'
// (format requires ALL JNs, not just a majority)
for (String jn : journalNodeHosts) {
  try (Socket s = new Socket()) {
    s.connect(new InetSocketAddress(jn, 8485), 2000);
  } catch (IOException e) {
    throw new IllegalStateException("JournalNode not reachable: " + jn, e);
  }
}

try {
  qjm.format(nsInfo, force);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Timed out waiting for format()")) {
    // identify the non-responding JN, fix/start it, then rerun format
    reportUnreachableJournalNodes();
    throw new RuntimeException("format failed: not all JournalNodes answered in time", e);
  }
  throw e;
}

When it happens

Trigger: Running hdfs namenode -format while any JournalNode in the quorum is down, firewalled on its RPC port (default 8485), or too slow to answer; dfs.qjm.operations.timeout set below JN response time.

Common situations: First bootstrap of an HA cluster before all JournalNodes are started; security groups blocking NN to JN traffic; JN disk or startup problems during initial formatting.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Timed out waiting for response from loggers
• Interrupted waiting for format() response
• Timed out waiting " + timeoutMs + "ms for a quorum of nodes
• Interrupted while determining if JNs have data
• Timed out waiting for doPreUpgrade() response

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c96ea9aa8b867a4d. Report an issue: GitHub.