vectordotdev/vector · info

record bytes should always be known

Error message

record bytes should always be known

What it means

For each acknowledged record marker with a known event count, the reader expect()s that the marker's byte-size payload (`data`) is present. Record markers are created in one place (reader.rs:521) always together with Some(record_bytes), so a Known-length marker without data cannot be constructed; the expect surfaces marker-construction regressions only.

Source

Thrown at lib/vector-buffers/src/variants/disk_v2/reader.rs:654

            {
                had_eligible_records = true;

                match len {
                    // Any marker with an assumed length implies a gap marker, which gets added
                    // automatically and represents a portion of the record ID range that was
                    // expected but missing. This is a long way of saying: we're missing records.
                    //
                    // We tally this up so that we can emit a single log event/set of metrics, as
                    // there may be many gap markers and emitting for each of them could be very noisy.
                    EligibleMarkerLength::Assumed(count) => {
                        events_skipped = events_skipped
                            .checked_add(count)
                            .expect("skipping more than 2^64 events at a time is obviously a bug");
                    }
                    // We got a valid marker representing a known number of events.
                    EligibleMarkerLength::Known(len) => {
                        // We specifically pass the size of the record, in bytes, as the marker data.
                        let record_bytes = data.expect("record bytes should always be known");

                        records_acknowledged = records_acknowledged.checked_add(1).expect(
                            "acknowledging more than 2^64 records at a time is obviously a bug",
                        );
                        events_acknowledged = events_acknowledged.checked_add(len).expect(
                            "acknowledging more than 2^64 events at a time is obviously a bug",
                        );
                        bytes_acknowledged = bytes_acknowledged.checked_add(record_bytes).expect(
                            "acknowledging more than 2^64 bytes at a time is obviously a bug",
                        );
                    }
                }
            }

            // We successfully processed at least one record, so update our buffer and ledger accounting.
            if had_eligible_records {
                self.ledger
                    .track_reads(events_acknowledged, bytes_acknowledged);

View on GitHub (pinned to 3708c39b12)

Solutions

1. No user action; if it fires, capture the backtrace and report it upstream

When it happens

Trigger: An EligibleMarker with EligibleMarkerLength::Known but data == None entering the record-acknowledgement loop — no code path constructs this shape; only an internal refactor bug or memory corruption trips it.

Common situations: None in practice; data-file markers (which legitimately carry None data) are consumed by a separate loop that never reads `data`.

Related errors

• record was already validated
• event count should never exceed u64
• Event count for a record cannot exceed 2^64 events.
• a record with a next ID must have an event count
• Ledger length cannot be greater than `u64`.

AI-assisted analysis of vectordotdev/vector@3708c39b12 (2026-08-20). Data as JSON: /api/errors/87e3e9c7d957029c. Report an issue: GitHub.