{"record":{"id":"e339677734eab59c","repo":"vectordotdev/vector","slug":"completing-more-than-2-16-data-files-at-a-time-is","errorCode":null,"errorMessage":"completing more than 2^16 data files at a time is obviously a bug","messagePattern":"completing more than 2\\^16 data files at a time is obviously a bug","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"lib/vector-buffers/src/variants/disk_v2/reader.rs","lineNumber":710,"sourceCode":"        //\n        // Alternatively, the core `next` logic may have just rolled over to a new data file, and\n        // we're seeing if we can fast track any eligible data file completions rather than waiting\n        // for more acknowledgements to come in.\n        let mut had_eligible_data_files = false;\n        let mut data_files_completed: u16 = 0;\n\n        if had_eligible_records || force_check_pending_data_files {\n            // Now handle data file completion.  We unconditionally check to see if any data files are\n            // eligible for logical completion, and process them immediately. Physical deletion is\n            // handled by the background cleanup task.\n\n            let mut completed_data_files = 0u16;\n            while let Some(EligibleMarker { .. }) = self.data_file_acks.get_next_eligible_marker() {\n                had_eligible_data_files = true;\n\n                completed_data_files = completed_data_files\n                    .checked_add(1)\n                    .expect(\"completing more than 2^16 data files at a time is obviously a bug\");\n            }\n\n            if had_eligible_data_files {\n                // Advance every logically completed file before flushing so the durable file\n                // checkpoint cannot lag behind the record checkpoint if we crash before deletion.\n                for _ in 0..completed_data_files {\n                    self.ledger.increment_acked_reader_file_id();\n                }\n                self.ledger.flush_reader_file_checkpoint()?;\n\n                data_files_completed = completed_data_files;\n            }\n        }\n\n        // If we managed to process any records _or_ any data file completions, we've made\n        // meaningful progress that writers may care about, so notify them.\n        if had_eligible_data_files || had_eligible_records {\n            self.ledger.notify_reader_waiters();","sourceCodeStart":692,"sourceCodeEnd":728,"githubUrl":"https://github.com/vectordotdev/vector/blob/3708c39b12a93212ed8b8d7510b4cc7769cb5864/lib/vector-buffers/src/variants/disk_v2/reader.rs#L692-L728","documentation":"After acknowledging records, the v2 disk reader logically completes fully-consumed data files, counting them per acknowledgement pass in a `u16`. More than 65,535 data files becoming eligible in a single pass overflows the counter and panics. With the default 128 MiB max data-file size this is unreachable, but a deliberately small `max_data_file_size` combined with a very large `max_buffer_size` can create enough data files to hit it.","triggerScenarios":"Configuring a disk_v2 buffer with max_data_file_size tiny relative to max_buffer_size (e.g. minimum file size with max_buffer_size in the hundreds of GiB), then letting acknowledgements fall far enough behind that a single acknowledgement pass completes more than 65,535 data files at once.","commonSituations":"Tuning experiments that shrink data files to force frequent rotation; soak tests with oversized buffer limits; configurations ported from disk_v1 assumptions where file counts scale differently.","solutions":["Raise `max_data_file_size` (or lower `max_buffer_size`) so max_buffer_size / max_data_file_size stays well below 65,536 files","Keep the default 128 MiB max_data_file_size unless there is a measured reason to change it","Keep acknowledgement lag bounded (avoid leaving the buffer unattended for very long periods) so completions spread across passes"],"exampleFix":"# before: can exceed 65536 files\n[sinks.out.buffer]\ntype = \"disk\"\nmax_data_file_size = 1048576        # 1 MiB\nmax_buffer_size = 137438953472      # 128 GiB\n\n# after: ~2048 files max\nmax_data_file_size = 67108864       # 64 MiB\nmax_buffer_size = 137438953472","handlingStrategy":"validation","validationCode":"// keep the worst-case file count far below u16::MAX\nfn file_count_within_bounds(max_buffer_size: u64, max_data_file_size: u64) -> bool {\n    max_data_file_size > 0 && (max_buffer_size / max_data_file_size) < 60_000\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Review max_buffer_size / max_data_file_size ratio whenever tuning either knob; keep it under ~60k files","Leave max_data_file_size at the 128 MiB default unless a measured workload justifies smaller files","In soak tests, exercise long acknowledgement lag explicitly so file-count extremes surface in testing, not production"],"tags":["rust","vector","disk-buffer","overflow","config","panic"],"backgroundTag":"arithmetic-overflow-panic","analyzedSha":"3708c39b12a93212ed8b8d7510b4cc7769cb5864","analyzedAt":"2026-08-20T07:02:18.786Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T20:17:18.057Z"}