apache/hadoop · error · IOException

Timed out waiting for response from loggers

Error message

Timed out waiting for response from loggers

What it means

hasSomeData() waits for all JournalNodes to report whether they hold edits, within dfs.qjm.operations.timeout. Any JournalNode that does not answer in time fails the check with this timeout, blocking the format-prompt path.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:278

      throw new IOException("Timed out waiting for format() response");
    }
    
    if (call.countExceptions() > 0) {
      call.rethrowException("Could not format one or more JournalNodes");
    }
  }

  @Override
  public boolean hasSomeData() throws IOException {
    QuorumCall<AsyncLogger, Boolean> call =
        loggers.isFormatted();

    try {
      call.waitFor(loggers.size(), 0, 0, timeoutMs, "hasSomeData");
    } catch (InterruptedException e) {
      throw new IOException("Interrupted while determining if JNs have data");
    } catch (TimeoutException e) {
      throw new IOException("Timed out waiting for response from loggers");
    }
    
    if (call.countExceptions() > 0) {
      call.rethrowException(
          "Unable to check if JNs are ready for formatting");
    }
    
    // If any of the loggers returned with a non-empty manifest, then
    // we should prompt for format.
    for (Boolean hasData : call.getResults().values()) {
      if (hasData) {
        return true;
      }
    }

    // Otherwise, none were formatted, we can safely format.
    return false;
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Ensure every JournalNode is running and reachable before formatting.
2. Raise dfs.qjm.operations.timeout and rerun if JournalNodes are merely slow to start.
3. Check JournalNode logs for errors answering the isFormatted RPC.

Example fix

<!-- raise the QJM operation timeout before formatting -->
<property>
  <name>dfs.qjm.operations.timeout</name>
  <value>120000</value>
</property>

// Preflight for 'hdfs namenode -format': every JN must answer isFormatted in time
static boolean journalNodesReady(List<InetSocketAddress> jns) {
  for (InetSocketAddress a : jns) {
    try (Socket s = new Socket()) {
      s.connect(a, 2000);
    } catch (IOException e) {
      return false; // this JN will time out hasSomeData()
    }
  }
  return true;
}

try {
  boolean hasData = qjm.hasSomeData();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Timed out waiting for response from loggers")) {
    // find and fix the silent JournalNode, then rerun the format flow
    reportUnreachableJournalNodes();
    throw new RuntimeException("hasSomeData timed out; JournalNode set unhealthy", e);
  }
  throw e;
}

When it happens

Trigger: One or more JournalNodes down, unreachable, or still starting when namenode -format checks for existing data; JN RPC port blocked from the NameNode host; dfs.qjm.operations.timeout too small relative to JN startup.

Common situations: HA cluster bootstrap with a JournalNode not yet started; firewall blocking port 8485; JN doing a long disk scan at startup while format runs.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Timed out waiting for format() response
• Interrupted waiting for format() response
• Timed out waiting " + timeoutMs + "ms for a quorum of nodes
• Interrupted while determining if JNs have data
• Timed out waiting for doPreUpgrade() response

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0724517581e0d363. Report an issue: GitHub.