{"record":{"id":"847630271897576c","repo":"apache/hadoop","slug":"we-expected-the-next-batch-of-events-to-start-with","errorCode":null,"errorMessage":"We expected the next batch of events to start with transaction ID {}, but it instead started with transaction ID {}. Most likely the intervening transactions were cleaned up as part of checkpointing.","messagePattern":"We expected the next batch of events to start with transaction ID (.+?), but it instead started with transaction ID (.+?)\\. Most likely the intervening transactions were cleaned up as part of checkpointing\\.","errorType":"exception","errorClass":"MissingEventsException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java","lineNumber":116,"sourceCode":"      // need to keep retrying until the NN sends us the latest committed txid\n      if (lastReadTxid == -1) {\n        LOG.debug(\"poll(): lastReadTxid is -1, reading current txid from NN\");\n        lastReadTxid = namenode.getCurrentEditLogTxid();\n        return null;\n      }\n      if (!it.hasNext()) {\n        EventBatchList el = namenode.getEditsFromTxid(lastReadTxid + 1);\n        if (el.getLastTxid() != -1) {\n          // we only want to set syncTxid when we were actually able to read some\n          // edits on the NN -- otherwise it will seem like edits are being\n          // generated faster than we can read them when the problem is really\n          // that we are temporarily unable to read edits\n          syncTxid = el.getSyncTxid();\n          it = el.getBatches().iterator();\n          long formerLastReadTxid = lastReadTxid;\n          lastReadTxid = el.getLastTxid();\n          if (el.getFirstTxid() != formerLastReadTxid + 1) {\n            throw new MissingEventsException(formerLastReadTxid + 1,\n                el.getFirstTxid());\n          }\n        } else {\n          LOG.debug(\"poll(): read no edits from the NN when requesting edits \" +\n              \"after txid {}\", lastReadTxid);\n          return null;\n        }\n      }\n\n      if (it.hasNext()) { // can be empty if el.getLastTxid != -1 but none of the\n        // newly seen edit log ops actually got converted to events\n        return it.next();\n      } else {\n        return null;\n      }\n    }\n  }\n","sourceCodeStart":98,"sourceCodeEnd":134,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java#L98-L134","documentation":"DFSInotifyEventInputStream tracks the last transaction id it read and, after each NameNode fetch, asserts the new batch's first txid equals lastReadTxid+1. If the NameNode has already purged those edits — checkpointing rolled them out of every retained edit segment — the gap can never be filled, and MissingEventsException reports the expected vs. actual first txid (analogous to a Kafka consumer whose committed offset falls outside the retained log).","triggerScenarios":"poll()/take() after the consumer was paused or down longer than the NN's retained-edit window; resuming from a persisted txid older than what checkpointing has cleaned (dfs.namenode.num.checkpoints.retained / dfs.namenode.num.extra.edits.retained too small); a consumer too slow to keep up with a busy namespace.","commonSituations":"Audit/metadata-sync daemons that fall behind on active clusters; inotify consumers restarted after a maintenance window; small test/staging NNs with aggressive checkpointing reused by long-lived consumers.","solutions":["Resync from the present: recreate the stream (getInotifyEventStream()) to start at the current txid, and backfill the missed window from a source of truth (full tree diff or snapshot comparison).","Persist the last-read txid after every processed batch (not just on shutdown) and resume promptly via getInotifyEventStream(lastReadTxid).","Increase edit retention on the NameNode (dfs.namenode.num.checkpoints.retained, dfs.namenode.num.extra.edits.retained) so it covers your worst-case consumer downtime.","Reduce consumer lag: batch the events, do less work per event, and alarm on (current txid - lastReadTxid) approaching the retention boundary."],"exampleFix":"// before\nEventBatch b = eis.poll();\n// MissingEventsException after a long consumer downtime\n\n// after\neis = dfs.getInotifyEventStream(); // restart from the current txid\n// + reconcile the missed window from a full listing/snapshot diff\ntry {\n  EventBatch b = eis.poll();\n} catch (MissingEventsException e) {\n  LOG.warn(\"edit-log gap; resyncing from current txid and reconciling\", e);\n  eis = dfs.getInotifyEventStream();\n}","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (MissingEventsException e) {\n  LOG.warn(\"edit-log gap; resuming from current txid\", e);\n  eis = dfs.getInotifyEventStream();      // restart from now\n  // schedule a full reconciliation (listing/snapshot diff) for the missed window\n}","preventionTips":["Persist the last-read txid after every processed batch, not only at shutdown, and resume via getInotifyEventStream(lastReadTxid).","Monitor consumer lag (current NN txid minus lastReadTxid) and alarm before it exceeds retained edits.","Size dfs.namenode.num.checkpoints.retained / dfs.namenode.num.extra.edits.retained to cover worst-case consumer downtime."],"tags":["hdfs","inotify","edit-log","txid-gap","event-stream","checkpoint","consumer-lag"],"backgroundTag":"offset-out-of-range","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}