apache/hadoop · error · IOException

Interrupted waiting for format() response

Error message

Interrupted waiting for format() response

What it means

QuorumJournalManager.format() sends format to every JournalNode and waits for all of them within dfs.qjm.operations.timeout. InterruptedException here means the formatting thread itself was cancelled (NN/tool shutdown), not that a JournalNode failed.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:258

    assert maxPromised >= 0;
    
    long myEpoch = maxPromised + 1;
    Map<AsyncLogger, NewEpochResponseProto> resps =
        loggers.waitForWriteQuorum(loggers.newEpoch(nsInfo, myEpoch),
            newEpochTimeoutMs, "newEpoch(" + myEpoch + ")");
        
    loggers.setEpoch(myEpoch);
    return resps;
  }
  
  @Override
  public void format(NamespaceInfo nsInfo, boolean force) throws IOException {
    QuorumCall<AsyncLogger, Void> call = loggers.format(nsInfo, force);
    try {
      call.waitFor(loggers.size(), loggers.size(), 0, timeoutMs,
          "format");
    } catch (InterruptedException e) {
      throw new IOException("Interrupted waiting for format() response");
    } catch (TimeoutException e) {
      throw new IOException("Timed out waiting for format() response");
    }
    
    if (call.countExceptions() > 0) {
      call.rethrowException("Could not format one or more JournalNodes");
    }
  }

  @Override
  public boolean hasSomeData() throws IOException {
    QuorumCall<AsyncLogger, Boolean> call =
        loggers.isFormatted();

    try {
      call.waitFor(loggers.size(), 0, 0, timeoutMs, "hasSomeData");
    } catch (InterruptedException e) {
      throw new IOException("Interrupted while determining if JNs have data");

View on GitHub (pinned to 2add963021)

Solutions

1. Identify what interrupted the formatter (shutdown hook, script kill) and rerun the format when it can run to completion.
2. Confirm all JournalNodes are up first so the rerun finishes within dfs.qjm.operations.timeout.
3. After an interrupted format, check each JN's formatted state before reusing the namespace, then re-run format -force if needed.

// Before formatting: confirm every JournalNode RPC port (default 8485) answers
static boolean allJournalNodesReachable(List<InetSocketAddress> jns) {
  for (InetSocketAddress a : jns) {
    try (Socket s = new Socket()) {
      s.connect(a, 2000);
    } catch (IOException e) {
      return false;
    }
  }
  return true;
}

try {
  qjm.format(nsInfo, force);
} catch (IOException e) {
  if (Thread.currentThread().isInterrupted()
      || (e.getMessage() != null && e.getMessage().contains("Interrupted waiting for format()"))) {
    Thread.currentThread().interrupt();
    // cancelled mid-format: verify JN formatted state, then rerun format -force
    verifyJournalNodeFormatState();
    throw new IllegalStateException("format cancelled; rerun after cleanup", e);
  }
  throw e;
}

When it happens

Trigger: hdfs namenode -format (or a QJM format API call) cancelled midway: JVM shutdown, Ctrl-C, or a caller-thread interrupt while waiting for JournalNode responses.

Common situations: Cluster bootstrap aborted by automation scripts; format step killed by a harness timeout; operators interrupting an HA format because one JN looked stuck.

Related errors

• Timed out waiting for format() response
• Interrupted while determining if JNs have data
• Timed out waiting for response from loggers
• Interrupted waiting " + timeoutMs + "ms for a quorum of node
• Interrupted waiting for doPreUpgrade() response

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c244f8215be2483e. Report an issue: GitHub.