apache/hadoop · warning · IOException

Did not get any valid JournaledEdits responses: " + msg

Error message

Did not get any valid JournaledEdits responses: " + msg

What it means

When tailing edits over RPC (in-progress tailing), the standby/observer NameNode asks JournalNodes for edits starting at a txn id via getJournaledEdits. A JournalNode that cannot serve the requested range from its journaled-edits cache answers with txnCount = -1 (cache miss). If the highest txn count among the quorum's responses is still negative, no JournalNode had usable edits and selectRpcInputStreams() throws. The caller catches this and falls back to HTTP streaming tailing.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:572

    List<Integer> responseCounts = new ArrayList<>();
    for (GetJournaledEditsResponseProto resp : responseMap.values()) {
      responseCounts.add(resp.getTxnCount());
    }
    Collections.sort(responseCounts);
    int highestTxnCount = responseCounts.get(responseCounts.size() - 1);
    if (LOG.isDebugEnabled() || highestTxnCount < 0) {
      StringBuilder msg = new StringBuilder("Requested edits starting from ");
      msg.append(fromTxnId).append("; got ").append(responseMap.size())
          .append(" responses: <");
      for (Map.Entry<AsyncLogger, GetJournaledEditsResponseProto> ent :
          responseMap.entrySet()) {
        msg.append("[").append(ent.getKey()).append(", ")
            .append(ent.getValue().getTxnCount()).append("],");
      }
      msg.append(">");
      if (highestTxnCount < 0) {
        throw new IOException("Did not get any valid JournaledEdits " +
            "responses: " + msg);
      } else {
        LOG.debug(msg.toString());
      }
    }
    // Cancel any outstanding calls to JN's.
    q.cancelCalls();

    int maxAllowedTxns = !onlyDurableTxns ? highestTxnCount :
        responseCounts.get(responseCounts.size() - loggers.getMajoritySize());
    if (maxAllowedTxns == 0) {
      LOG.debug("No new edits available in logs; requested starting from ID {}",
          fromTxnId);
      return;
    }
    LogAction logAction = selectInputStreamLogHelper.record(fromTxnId);
    if (logAction.shouldLog()) {
      LOG.info("Selected loggers with >= " + maxAllowedTxns + " transactions " +

View on GitHub (pinned to 2add963021)

Solutions

1. Read the response dump in the exception message: it lists each JournalNode's txnCount for the requested fromTxnId.
2. No immediate action if fallback works — the NN logs 'falling back to streaming' and continues over HTTP.
3. If JournalMetrics rpcRequestCacheMissAmount shows chronic misses, grow the edit cache (dfs.journalnode.edit-cache.size / dfs.journalnode.edit-cache.size.fraction) on the JournalNodes.
4. If the standby is hopelessly behind, run 'hdfs namenode -bootstrapStandby' or take a checkpoint instead of tailing from a stale point.

Example fix

<!-- on each JournalNode: enlarge the edit cache window -->
<property>
  <name>dfs.journalnode.edit-cache.size.fraction</name>
  <value>0.5</value>
</property>

try {
  selectRpcInputStreams(streams, fromTxnId, onlyDurableTxns);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Did not get any valid JournaledEdits")) {
    // all JournalNodes missed the range in their edit cache:
    // fall back to HTTP streaming tailing (as QJM itself does)
    LOG.warn("RPC tailing cache miss; falling back to HTTP streaming", e);
    selectStreamingInputStreams(streams, fromTxnId, inProgressOk, onlyDurableTxns);
    return;
  }
  throw e;
}

When it happens

Trigger: Every responding JournalNode missed the requested range in its edit cache: the requested fromTxnId was already evicted (dfs.journalnode.edit-cache.size too small for the edit rate), the cache was reset after the JN lagged or restarted, or tailing starts behind the cache's first retained txn.

Common situations: Standby/Observer NameNode lagging far behind the active; JournalNode restart clearing the cache; edit cache sized via dfs.journalnode.edit-cache.size.fraction too small for bursty edit traffic; usually transient at tailing start and self-heals via fallback.

Related errors

• Interrupted waiting " + timeoutMs + "ms for a quorum of node
• Timed out waiting " + timeoutMs + "ms for a quorum of nodes
• Journal disabled until next roll
• Interrupted waiting for format() response
• Timed out waiting for format() response

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/19fec30b1f3a08c2. Report an issue: GitHub.