vectordotdev/vector · info

skipping more than 2^64 events at a time is obviously a bug

Error message

skipping more than 2^64 events at a time is obviously a bug

What it means

While acknowledging records in the v2 disk reader, gap markers (record-ID ranges that were expected but missing) are tallied into a u64 accumulator via checked_add. Overflow would require more than 2^64 skipped events within one acknowledgement pass — more events than the entire u64 record-ID space allows — so the expect marks a logical impossibility whose only trigger is an internal ID-accounting bug.

Source

Thrown at lib/vector-buffers/src/variants/disk_v2/reader.rs:649

        if consumed_acks > 0 {
            self.record_acks.add_acknowledgements(consumed_acks);

            while let Some(EligibleMarker { len, data, .. }) =
                self.record_acks.get_next_eligible_marker()
            {
                had_eligible_records = true;

                match len {
                    // Any marker with an assumed length implies a gap marker, which gets added
                    // automatically and represents a portion of the record ID range that was
                    // expected but missing. This is a long way of saying: we're missing records.
                    //
                    // We tally this up so that we can emit a single log event/set of metrics, as
                    // there may be many gap markers and emitting for each of them could be very noisy.
                    EligibleMarkerLength::Assumed(count) => {
                        events_skipped = events_skipped
                            .checked_add(count)
                            .expect("skipping more than 2^64 events at a time is obviously a bug");
                    }
                    // We got a valid marker representing a known number of events.
                    EligibleMarkerLength::Known(len) => {
                        // We specifically pass the size of the record, in bytes, as the marker data.
                        let record_bytes = data.expect("record bytes should always be known");

                        records_acknowledged = records_acknowledged.checked_add(1).expect(
                            "acknowledging more than 2^64 records at a time is obviously a bug",
                        );
                        events_acknowledged = events_acknowledged.checked_add(len).expect(
                            "acknowledging more than 2^64 events at a time is obviously a bug",
                        );
                        bytes_acknowledged = bytes_acknowledged.checked_add(record_bytes).expect(
                            "acknowledging more than 2^64 bytes at a time is obviously a bug",
                        );
                    }
                }
            }

View on GitHub (pinned to 3708c39b12)

Solutions

1. Capture the backtrace (RUST_BACKTRACE=1) and report upstream
2. Move the buffer data directory aside to restore service while the bug is investigated

When it happens

Trigger: Accumulating more than u64::MAX skipped events in a single handle_pending_acknowledgements run; record IDs are u64 and bounded, so no buffer contents or acknowledgment pattern can legitimately reach it.

Common situations: None in practice; would indicate a record-ID arithmetic bug in Vector rather than an operational condition.

Related errors

• record was already validated
• event count should never exceed u64
• Event count for a record cannot exceed 2^64 events.
• a record with a next ID must have an event count
• Ledger length cannot be greater than `u64`.

AI-assisted analysis of vectordotdev/vector@3708c39b12 (2026-08-20). Data as JSON: /api/errors/5db71811760b4db0. Report an issue: GitHub.