apache/hadoop · critical · NewerTxnIdException

Highest txn ID available in the journal is %d, but requested

Error message

Highest txn ID available in the journal is %d, but requested txns starting at %d.

What it means

Journal.getJournaledEdits throws NewerTxnIdException (an IOException) when the requested sinceTxId is strictly greater than highestTxId + 1 — the journal is missing edits that should already exist. This is not a 'wait for writes' case (that returns a 0-count response); it means the journal has permanently lagged behind, e.g., it skipped a range. Per HDFS-16659/HDFS-16771 the NameNode must treat it as an error rather than a gap-fill, because silently skipping missing transactions would corrupt NN state.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:767

      throw new IOException("The journal edits cache is not enabled, which " +
          "is a requirement to fetch journaled edits via RPC. Please enable " +
          "it via " + DFSConfigKeys.DFS_HA_TAILEDITS_INPROGRESS_KEY);
    }
    long highestTxId = getHighestWrittenTxId();
    if (sinceTxId == highestTxId + 1) {
      // Requested edits that don't exist yet, but this is expected,
      // because namenode always get the journaled edits with the sinceTxId
      // equal to image.getLastAppliedTxId() + 1. Short-circuiting the cache here
      // and returning a response with a count of 0.
      metrics.rpcEmptyResponses.incr();
      return GetJournaledEditsResponseProto.newBuilder().setTxnCount(0).build();
    } else if (sinceTxId > highestTxId + 1) {
      // Requested edits that don't exist yet and this is unexpected. Means that there is a lag
      // in this journal that does not contain some edits that should exist.
      // Throw one NewerTxnIdException to make namenode treat this response as an exception.
      // More detailed info please refer to: HDFS-16659 and HDFS-16771.
      metrics.rpcEmptyResponses.incr();
      throw new NewerTxnIdException(
          "Highest txn ID available in the journal is %d, but requested txns starting at %d.",
          highestTxId, sinceTxId);
    }
    try {
      List<ByteBuffer> buffers = new ArrayList<>();
      int txnCount = cache.retrieveEdits(sinceTxId, maxTxns, buffers);
      int totalSize = 0;
      for (ByteBuffer buf : buffers) {
        totalSize += buf.remaining();
      }
      metrics.txnsServedViaRpc.incr(txnCount);
      metrics.bytesServedViaRpc.incr(totalSize);
      ByteString.Output output = ByteString.newOutput(totalSize);
      for (ByteBuffer buf : buffers) {
        output.write(buf.array(), buf.position(), buf.remaining());
      }
      return GetJournaledEditsResponseProto.newBuilder()
          .setTxnCount(txnCount)

View on GitHub (pinned to 2add963021)

Solutions

1. Resync the lagging JournalNode: stop it, clear/verify its dfs.journalnode.edits.dir for the jid, restart, and let dfs.journalnode.enable.sync refill it from peer journals (or bootstrap it manually from a healthy journal).
2. Confirm which journal is behind by comparing getHighestWrittenTxId across JournalNodes (logs/JMX) — the thrower is by definition the behind one.
3. On the NameNode side the exception is handled by failing the tailing attempt; a failover/fresh tail once the journal is resynced clears the condition — do not disable the check.
4. Enable dfs.journalnode.enable.sync=true on JournalNodes so any journal that misses writes tails them from peers automatically and never drifts.

Example fix

# before
# NewerTxnIdException: Highest txn ID available in the journal is 5200, but requested txns starting at 5350.

# after: resync the lagging journal from peers
# 1) identify the lagging JN (the one throwing), stop it
# 2) keep/repair its storage, enable edit sync, restart
<property>
  <name>dfs.journalnode.enable.sync</name>
  <value>true</value>
</property>
# the JN tails 5201..5349 from peers; NameNode tailing then succeeds

// Compare positions before tailing
long highest = journal.getHighestWrittenTxId(); // via manifest / committedTxnId
if (sinceTxId > highest + 1) {
  // journal is behind: fetch missing edits from another journal/NN instead
}

try {
  resp = jn.getJournaledEdits(sinceTxId, maxTxns);
} catch (NewerTxnIdException e) {
  // this journal permanently lacks edits [highest+1, sinceTxId-1]:
  // source those transactions from a healthy journal or shared edits, then resync this journal
  edits = fetchFromHealthyJournal(sinceTxId, maxTxns);
  scheduleJournalResync(journalId);
}

When it happens

Trigger: getJournaledEdits(sinceTxId=N, ...) with N > highestWrittenTxId()+1: the tailer's position is ahead of this journal's last written txid, so the journal never saw some transactions that other journals already committed. Occurs when one JournalNode lost/rerolled segments, was reformatted, or fell out of the quorum write path.

Common situations: A JournalNode was down during part of the write stream and never resynced; journal dir wiped/restored from stale backup; mixed epochs after failover where one journal accepted a different segment layout; NameNode tailing in-progress edits hits the lagging journal before journal-node sync catches up.

Related errors

• No log file to finalize at transaction ID {} ; journal id: {
• The journal edits cache is not enabled, which is a requireme
• The log file {} seems to contain valid transactions ; journa
• {} does not support method msync
• Not able to initialize fs in getDefaultBlockSize for path <

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a50aaddd3abf515b. Report an issue: GitHub.