vectordotdev/vector · error

completing more than 2^16 data files at a time is obviously

Error message

completing more than 2^16 data files at a time is obviously a bug

What it means

After acknowledging records, the v2 disk reader logically completes fully-consumed data files, counting them per acknowledgement pass in a `u16`. More than 65,535 data files becoming eligible in a single pass overflows the counter and panics. With the default 128 MiB max data-file size this is unreachable, but a deliberately small `max_data_file_size` combined with a very large `max_buffer_size` can create enough data files to hit it.

Source

Thrown at lib/vector-buffers/src/variants/disk_v2/reader.rs:710

        //
        // Alternatively, the core `next` logic may have just rolled over to a new data file, and
        // we're seeing if we can fast track any eligible data file completions rather than waiting
        // for more acknowledgements to come in.
        let mut had_eligible_data_files = false;
        let mut data_files_completed: u16 = 0;

        if had_eligible_records || force_check_pending_data_files {
            // Now handle data file completion.  We unconditionally check to see if any data files are
            // eligible for logical completion, and process them immediately. Physical deletion is
            // handled by the background cleanup task.

            let mut completed_data_files = 0u16;
            while let Some(EligibleMarker { .. }) = self.data_file_acks.get_next_eligible_marker() {
                had_eligible_data_files = true;

                completed_data_files = completed_data_files
                    .checked_add(1)
                    .expect("completing more than 2^16 data files at a time is obviously a bug");
            }

            if had_eligible_data_files {
                // Advance every logically completed file before flushing so the durable file
                // checkpoint cannot lag behind the record checkpoint if we crash before deletion.
                for _ in 0..completed_data_files {
                    self.ledger.increment_acked_reader_file_id();
                }
                self.ledger.flush_reader_file_checkpoint()?;

                data_files_completed = completed_data_files;
            }
        }

        // If we managed to process any records _or_ any data file completions, we've made
        // meaningful progress that writers may care about, so notify them.
        if had_eligible_data_files || had_eligible_records {
            self.ledger.notify_reader_waiters();

View on GitHub (pinned to 3708c39b12)

Solutions

1. Raise `max_data_file_size` (or lower `max_buffer_size`) so max_buffer_size / max_data_file_size stays well below 65,536 files
2. Keep the default 128 MiB max_data_file_size unless there is a measured reason to change it
3. Keep acknowledgement lag bounded (avoid leaving the buffer unattended for very long periods) so completions spread across passes

Example fix

# before: can exceed 65536 files
[sinks.out.buffer]
type = "disk"
max_data_file_size = 1048576        # 1 MiB
max_buffer_size = 137438953472      # 128 GiB

# after: ~2048 files max
max_data_file_size = 67108864       # 64 MiB
max_buffer_size = 137438953472

// keep the worst-case file count far below u16::MAX
fn file_count_within_bounds(max_buffer_size: u64, max_data_file_size: u64) -> bool {
    max_data_file_size > 0 && (max_buffer_size / max_data_file_size) < 60_000
}

When it happens

Trigger: Configuring a disk_v2 buffer with max_data_file_size tiny relative to max_buffer_size (e.g. minimum file size with max_buffer_size in the hundreds of GiB), then letting acknowledgements fall far enough behind that a single acknowledgement pass completes more than 65,535 data files at once.

Common situations: Tuning experiments that shrink data files to force frequent rotation; soak tests with oversized buffer limits; configurations ported from disk_v1 assumptions where file counts scale differently.

Related errors

• skipping more than 2^64 events at a time is obviously a bug
• Reader encountered unrecoverable error: {e:?}
• Timestamp out of range
• failed to read glob pattern
• effective reader file ID must be in the checkpoint window

AI-assisted analysis of vectordotdev/vector@3708c39b12 (2026-08-20). Data as JSON: /api/errors/e339677734eab59c. Report an issue: GitHub.