apache/hadoop · error · MissingEventsException

We expected the next batch of events to start with transacti

Error message

We expected the next batch of events to start with transaction ID {}, but it instead started with transaction ID {}. Most likely the intervening transactions were cleaned up as part of checkpointing.

What it means

DFSInotifyEventInputStream tracks the last transaction id it read and, after each NameNode fetch, asserts the new batch's first txid equals lastReadTxid+1. If the NameNode has already purged those edits — checkpointing rolled them out of every retained edit segment — the gap can never be filled, and MissingEventsException reports the expected vs. actual first txid (analogous to a Kafka consumer whose committed offset falls outside the retained log).

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java:116

      // need to keep retrying until the NN sends us the latest committed txid
      if (lastReadTxid == -1) {
        LOG.debug("poll(): lastReadTxid is -1, reading current txid from NN");
        lastReadTxid = namenode.getCurrentEditLogTxid();
        return null;
      }
      if (!it.hasNext()) {
        EventBatchList el = namenode.getEditsFromTxid(lastReadTxid + 1);
        if (el.getLastTxid() != -1) {
          // we only want to set syncTxid when we were actually able to read some
          // edits on the NN -- otherwise it will seem like edits are being
          // generated faster than we can read them when the problem is really
          // that we are temporarily unable to read edits
          syncTxid = el.getSyncTxid();
          it = el.getBatches().iterator();
          long formerLastReadTxid = lastReadTxid;
          lastReadTxid = el.getLastTxid();
          if (el.getFirstTxid() != formerLastReadTxid + 1) {
            throw new MissingEventsException(formerLastReadTxid + 1,
                el.getFirstTxid());
          }
        } else {
          LOG.debug("poll(): read no edits from the NN when requesting edits " +
              "after txid {}", lastReadTxid);
          return null;
        }
      }

      if (it.hasNext()) { // can be empty if el.getLastTxid != -1 but none of the
        // newly seen edit log ops actually got converted to events
        return it.next();
      } else {
        return null;
      }
    }
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Resync from the present: recreate the stream (getInotifyEventStream()) to start at the current txid, and backfill the missed window from a source of truth (full tree diff or snapshot comparison).
2. Persist the last-read txid after every processed batch (not just on shutdown) and resume promptly via getInotifyEventStream(lastReadTxid).
3. Increase edit retention on the NameNode (dfs.namenode.num.checkpoints.retained, dfs.namenode.num.extra.edits.retained) so it covers your worst-case consumer downtime.
4. Reduce consumer lag: batch the events, do less work per event, and alarm on (current txid - lastReadTxid) approaching the retention boundary.

Example fix

// before
EventBatch b = eis.poll();
// MissingEventsException after a long consumer downtime

// after
eis = dfs.getInotifyEventStream(); // restart from the current txid
// + reconcile the missed window from a full listing/snapshot diff
try {
  EventBatch b = eis.poll();
} catch (MissingEventsException e) {
  LOG.warn("edit-log gap; resyncing from current txid and reconciling", e);
  eis = dfs.getInotifyEventStream();
}

catch (MissingEventsException e) {
  LOG.warn("edit-log gap; resuming from current txid", e);
  eis = dfs.getInotifyEventStream();      // restart from now
  // schedule a full reconciliation (listing/snapshot diff) for the missed window
}

When it happens

Trigger: poll()/take() after the consumer was paused or down longer than the NN's retained-edit window; resuming from a persisted txid older than what checkpointing has cleaned (dfs.namenode.num.checkpoints.retained / dfs.namenode.num.extra.edits.retained too small); a consumer too slow to keep up with a busy namespace.

Common situations: Audit/metadata-sync daemons that fall behind on active clusters; inotify consumers restarted after a maintenance window; small test/staging NNs with aggressive checkpointing reused by long-lived consumers.

Related errors

• No logs to roll forward from lastApplied
• The length of the feature flag section was negative at {} by
• Interrupted waiting " + timeoutMs + "ms for a quorum of node
• Timed out waiting " + timeoutMs + "ms for a quorum of nodes
• Journal disabled until next roll

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/847630271897576c. Report an issue: GitHub.