{"record":{"id":"a50aaddd3abf515b","repo":"apache/hadoop","slug":"highest-txn-id-available-in-the-journal-is-d-but","errorCode":null,"errorMessage":"Highest txn ID available in the journal is %d, but requested txns starting at %d.","messagePattern":"Highest txn ID available in the journal is (.+?), but requested txns starting at (.+?)\\.","errorType":"exception","errorClass":"NewerTxnIdException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java","lineNumber":767,"sourceCode":" throw new IOException(\"The journal edits cache is not enabled, which \" +\n \"is a requirement to fetch journaled edits via RPC. Please enable \" +\n \"it via \" + DFSConfigKeys.DFS_HA_TAILEDITS_INPROGRESS_KEY);\n }\n long highestTxId = getHighestWrittenTxId();\n if (sinceTxId == highestTxId + 1) {\n // Requested edits that don't exist yet, but this is expected,\n // because namenode always get the journaled edits with the sinceTxId\n // equal to image.getLastAppliedTxId() + 1. Short-circuiting the cache here\n // and returning a response with a count of 0.\n metrics.rpcEmptyResponses.incr();\n return GetJournaledEditsResponseProto.newBuilder().setTxnCount(0).build();\n } else if (sinceTxId > highestTxId + 1) {\n // Requested edits that don't exist yet and this is unexpected. Means that there is a lag\n // in this journal that does not contain some edits that should exist.\n // Throw one NewerTxnIdException to make namenode treat this response as an exception.\n // More detailed info please refer to: HDFS-16659 and HDFS-16771.\n metrics.rpcEmptyResponses.incr();\n throw new NewerTxnIdException(\n \"Highest txn ID available in the journal is %d, but requested txns starting at %d.\",\n highestTxId, sinceTxId);\n }\n try {\n List buffers = new ArrayList<>();\n int txnCount = cache.retrieveEdits(sinceTxId, maxTxns, buffers);\n int totalSize = 0;\n for (ByteBuffer buf : buffers) {\n totalSize += buf.remaining();\n }\n metrics.txnsServedViaRpc.incr(txnCount);\n metrics.bytesServedViaRpc.incr(totalSize);\n ByteString.Output output = ByteString.newOutput(totalSize);\n for (ByteBuffer buf : buffers) {\n output.write(buf.array(), buf.position(), buf.remaining());\n }\n return GetJournaledEditsResponseProto.newBuilder()\n .setTxnCount(txnCount)","sourceCodeStart":749,"sourceCodeEnd":785,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java#L749-L785","documentation":"Journal.getJournaledEdits throws NewerTxnIdException (an IOException) when the requested sinceTxId is strictly greater than highestTxId + 1 — the journal is missing edits that should already exist. This is not a 'wait for writes' case (that returns a 0-count response); it means the journal has permanently lagged behind, e.g., it skipped a range. Per HDFS-16659/HDFS-16771 the NameNode must treat it as an error rather than a gap-fill, because silently skipping missing transactions would corrupt NN state.","triggerScenarios":"getJournaledEdits(sinceTxId=N, ...) with N > highestWrittenTxId()+1: the tailer's position is ahead of this journal's last written txid, so the journal never saw some transactions that other journals already committed. Occurs when one JournalNode lost/rerolled segments, was reformatted, or fell out of the quorum write path.","commonSituations":"A JournalNode was down during part of the write stream and never resynced; journal dir wiped/restored from stale backup; mixed epochs after failover where one journal accepted a different segment layout; NameNode tailing in-progress edits hits the lagging journal before journal-node sync catches up.","solutions":["Resync the lagging JournalNode: stop it, clear/verify its dfs.journalnode.edits.dir for the jid, restart, and let dfs.journalnode.enable.sync refill it from peer journals (or bootstrap it manually from a healthy journal).","Confirm which journal is behind by comparing getHighestWrittenTxId across JournalNodes (logs/JMX) — the thrower is by definition the behind one.","On the NameNode side the exception is handled by failing the tailing attempt; a failover/fresh tail once the journal is resynced clears the condition — do not disable the check.","Enable dfs.journalnode.enable.sync=true on JournalNodes so any journal that misses writes tails them from peers automatically and never drifts."],"exampleFix":"# before\n# NewerTxnIdException: Highest txn ID available in the journal is 5200, but requested txns starting at 5350.\n\n# after: resync the lagging journal from peers\n# 1) identify the lagging JN (the one throwing), stop it\n# 2) keep/repair its storage, enable edit sync, restart\n\n dfs.journalnode.enable.sync\n true\n\n# the JN tails 5201..5349 from peers; NameNode tailing then succeeds","handlingStrategy":"fallback","validationCode":"// Compare positions before tailing\nlong highest = journal.getHighestWrittenTxId(); // via manifest / committedTxnId\nif (sinceTxId > highest + 1) {\n // journal is behind: fetch missing edits from another journal/NN instead\n}","typeGuard":null,"tryCatchPattern":"try {\n resp = jn.getJournaledEdits(sinceTxId, maxTxns);\n} catch (NewerTxnIdException e) {\n // this journal permanently lacks edits [highest+1, sinceTxId-1]:\n // source those transactions from a healthy journal or shared edits, then resync this journal\n edits = fetchFromHealthyJournal(sinceTxId, maxTxns);\n scheduleJournalResync(journalId);\n}","preventionTips":["Enable dfs.journalnode.enable.sync so journals never drift after outages.","Monitor per-journal highestWrittenTxId and committedTxnId; alert on divergence beyond a threshold.","After any JournalNode outage, verify it caught up before the NN tails from it."],"tags":["hdfs","qjournal","journal-lag","ha","tail-edits","data-loss"],"backgroundTag":"journal-replica-lag","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}