vectordotdev/vector · info

should not fail to add marker for data file completion

Error message

should not fail to add marker for data file completion

What it means

When the v2 disk reader finishes a data file, it appends a completion marker to its in-memory acknowledgement ledger at the current record offset. `add_marker` fails when the new marker would not follow the existing ones (IDs out of order), which cannot happen because data_file_marked_record_count advances monotonically with each roll; the expect surfaces internal offset-accounting regressions only.

Source

Thrown at lib/vector-buffers/src/variants/disk_v2/reader.rs:782

        let bytes_read = self.bytes_read;

        debug!(
            data_file_path = data_file_path.to_string_lossy().as_ref(),
            first_record_id = data_file_start_record_id,
            last_record_id = self.last_reader_record_id,
            record_count = data_file_record_count,
            event_count = data_file_event_count,
            bytes_read,
            "Marking data file for completion."
        );

        let data_file_marker_id = self.data_file_marked_record_count;
        self.data_file_marked_record_count += data_file_record_count;
        self.data_file_record_count = 0;

        self.data_file_acks
            .add_marker(data_file_marker_id, Some(data_file_record_count), None)
            .expect("should not fail to add marker for data file completion");

        // Now reset our internal state so we can go for the next data file.
        self.reset();
        self.ledger.increment_unacked_reader_file_id();

        debug!("Rolling to next data file.");
    }

    /// Ensures this reader is ready to attempt reading the next record.
    #[cfg_attr(test, instrument(skip(self), level = "debug"))]
    async fn ensure_ready_for_read(&mut self) -> io::Result<()> {
        // We have nothing to do if we already have a data file open.
        if self.reader.is_some() {
            return Ok(());
        }

        // Try to open the current reader data file.  This might not _yet_ exist, in which case
        // we'll simply wait for the writer to signal to us that progress has been made, which

View on GitHub (pinned to 3708c39b12)

Solutions

1. No user action; if it fires, capture RUST_BACKTRACE=1 output and report it upstream with the Vector version

When it happens

Trigger: roll_to_next_data_file adding a marker whose ID is not greater than the last marker's — no external input, config, or on-disk state influences these in-memory counters; only a code regression trips it.

Common situations: None in practice; marker state is process-local and sequential by construction.

Related errors

• record was already validated
• event count should never exceed u64
• Event count for a record cannot exceed 2^64 events.
• a record with a next ID must have an event count
• Ledger length cannot be greater than `u64`.

AI-assisted analysis of vectordotdev/vector@3708c39b12 (2026-08-20). Data as JSON: /api/errors/2067242351196c66. Report an issue: GitHub.