{"record":{"id":"19fec30b1f3a08c2","repo":"apache/hadoop","slug":"did-not-get-any-valid-journalededits-responses","errorCode":null,"errorMessage":"Did not get any valid JournaledEdits responses: \" + msg","messagePattern":"Did not get any valid JournaledEdits responses: \" \\+ msg","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java","lineNumber":572,"sourceCode":"\n    List<Integer> responseCounts = new ArrayList<>();\n    for (GetJournaledEditsResponseProto resp : responseMap.values()) {\n      responseCounts.add(resp.getTxnCount());\n    }\n    Collections.sort(responseCounts);\n    int highestTxnCount = responseCounts.get(responseCounts.size() - 1);\n    if (LOG.isDebugEnabled() || highestTxnCount < 0) {\n      StringBuilder msg = new StringBuilder(\"Requested edits starting from \");\n      msg.append(fromTxnId).append(\"; got \").append(responseMap.size())\n          .append(\" responses: <\");\n      for (Map.Entry<AsyncLogger, GetJournaledEditsResponseProto> ent :\n          responseMap.entrySet()) {\n        msg.append(\"[\").append(ent.getKey()).append(\", \")\n            .append(ent.getValue().getTxnCount()).append(\"],\");\n      }\n      msg.append(\">\");\n      if (highestTxnCount < 0) {\n        throw new IOException(\"Did not get any valid JournaledEdits \" +\n            \"responses: \" + msg);\n      } else {\n        LOG.debug(msg.toString());\n      }\n    }\n    // Cancel any outstanding calls to JN's.\n    q.cancelCalls();\n\n    int maxAllowedTxns = !onlyDurableTxns ? highestTxnCount :\n        responseCounts.get(responseCounts.size() - loggers.getMajoritySize());\n    if (maxAllowedTxns == 0) {\n      LOG.debug(\"No new edits available in logs; requested starting from ID {}\",\n          fromTxnId);\n      return;\n    }\n    LogAction logAction = selectInputStreamLogHelper.record(fromTxnId);\n    if (logAction.shouldLog()) {\n      LOG.info(\"Selected loggers with >= \" + maxAllowedTxns + \" transactions \" +","sourceCodeStart":554,"sourceCodeEnd":590,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java#L554-L590","documentation":"When tailing edits over RPC (in-progress tailing), the standby/observer NameNode asks JournalNodes for edits starting at a txn id via getJournaledEdits. A JournalNode that cannot serve the requested range from its journaled-edits cache answers with txnCount = -1 (cache miss). If the highest txn count among the quorum's responses is still negative, no JournalNode had usable edits and selectRpcInputStreams() throws. The caller catches this and falls back to HTTP streaming tailing.","triggerScenarios":"Every responding JournalNode missed the requested range in its edit cache: the requested fromTxnId was already evicted (dfs.journalnode.edit-cache.size too small for the edit rate), the cache was reset after the JN lagged or restarted, or tailing starts behind the cache's first retained txn.","commonSituations":"Standby/Observer NameNode lagging far behind the active; JournalNode restart clearing the cache; edit cache sized via dfs.journalnode.edit-cache.size.fraction too small for bursty edit traffic; usually transient at tailing start and self-heals via fallback.","solutions":["Read the response dump in the exception message: it lists each JournalNode's txnCount for the requested fromTxnId.","No immediate action if fallback works — the NN logs 'falling back to streaming' and continues over HTTP.","If JournalMetrics rpcRequestCacheMissAmount shows chronic misses, grow the edit cache (dfs.journalnode.edit-cache.size / dfs.journalnode.edit-cache.size.fraction) on the JournalNodes.","If the standby is hopelessly behind, run 'hdfs namenode -bootstrapStandby' or take a checkpoint instead of tailing from a stale point."],"exampleFix":"<!-- on each JournalNode: enlarge the edit cache window -->\n<property>\n  <name>dfs.journalnode.edit-cache.size.fraction</name>\n  <value>0.5</value>\n</property>","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  selectRpcInputStreams(streams, fromTxnId, onlyDurableTxns);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Did not get any valid JournaledEdits\")) {\n    // all JournalNodes missed the range in their edit cache:\n    // fall back to HTTP streaming tailing (as QJM itself does)\n    LOG.warn(\"RPC tailing cache miss; falling back to HTTP streaming\", e);\n    selectStreamingInputStreams(streams, fromTxnId, inProgressOk, onlyDurableTxns);\n    return;\n  }\n  throw e;\n}","preventionTips":["Monitor JournalNode JournalMetrics rpcRequestCacheMissAmount; resize the edit cache when it grows.","Size dfs.journalnode.edit-cache.size / .fraction to your peak edit rate times expected standby lag.","Avoid letting standbys/observers lag far behind; checkpoint or bootstrap-standby when lag is large.","Expect occasional misses right after JournalNode restarts (cache is cold) — the HTTP fallback covers them."],"tags":["hdfs","qjm","journal-node","standby","edit-tailing","cache-miss","rpc"],"backgroundTag":"quorum-no-valid-responses","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}