{"record":{"id":"6a9d0ba77aa0f328","repo":"vectordotdev/vector","slug":"reader-encountered-unrecoverable-error-e","errorCode":null,"errorMessage":"Reader encountered unrecoverable error: {e:?}","messagePattern":"Reader encountered unrecoverable error: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"lib/vector-buffers/src/topology/channel/receiver.rs","lineNumber":59,"sourceCode":"}\n\nimpl<T> ReceiverAdapter<T>\nwhere\n    T: Bufferable,\n{\n    pub(crate) async fn next(&mut self) -> Option<T> {\n        match self {\n            ReceiverAdapter::InMemory(rx) => rx.next().await,\n            ReceiverAdapter::DiskV2(reader) => loop {\n                match reader.next().await {\n                    Ok(result) => break result,\n                    Err(e) => match e.as_recoverable_error() {\n                        Some(re) => {\n                            // If we've hit a recoverable error, we'll emit an event to indicate as much but we'll still\n                            // keep trying to read the next available record.\n                            emit(re);\n                        }\n                        None => panic!(\"Reader encountered unrecoverable error: {e:?}\"),\n                    },\n                }\n            },\n        }\n    }\n}\n\n/// A buffer receiver.\n///\n/// The receiver handles retrieving events from the buffer, regardless of the overall buffer configuration.\n///\n/// If a buffer was configured to operate in \"overflow\" mode, then the receiver will be responsible\n/// for querying the overflow buffer as well.  The ordering of events when operating in \"overflow\"\n/// is undefined, as the receiver will try to manage polling both its own buffer, as well as the\n/// overflow buffer, in order to fairly balance throughput.\n#[derive(Debug)]\npub struct BufferReceiver<T: Bufferable> {\n    base: ReceiverAdapter<T>,","sourceCodeStart":41,"sourceCodeEnd":77,"githubUrl":"https://github.com/vectordotdev/vector/blob/3708c39b12a93212ed8b8d7510b4cc7769cb5864/lib/vector-buffers/src/topology/channel/receiver.rs#L41-L77","documentation":"The disk-v2 buffer reader inside `ReceiverAdapter::next` (lib/vector-buffers/src/topology/channel/receiver.rs:47-61) loops over `reader.next().await` and classifies each `Err`: recoverable errors are emitted as internal events and reading continues, but an error with no recoverable variant is considered fatal buffer corruption and panics, tearing down the topology task. This is intentional fail-fast behavior — the on-disk ledger/segment data is untrusted and cannot be safely skipped. Recoverable errors surfaced earlier (e.g. `BufferReadError` events) are the warning sign that disk reads are degraded.","triggerScenarios":"A Vector deployment configured with `buffers.type = \"disk\"` (disk v2, `data_dir` on persistent storage) where `BufferReader::next` returns a non-recoverable error: corrupted or truncated ledger/segment files after an unclean shutdown, unreadable data dir (permissions, failing disk, ENOSPC), or buffer files written by an incompatible Vector version.","commonSituations":"Hard power loss or OOM-kill while the disk buffer held data; running out of disk space under sustained backpressure; downgrading or skipping Vector versions across restarts so the on-disk format no longer parses; copying a `data_dir` between machines/versions; container deployments where the volume backing `data_dir` is remounted read-only.","solutions":["Check the obvious environmental causes first: free space on the `data_dir` volume, directory permissions, and disk/filesystem health (dmesg/SMART for I/O errors).","Restore version compatibility: restart with the same (or a format-compatible, per release notes) Vector version that wrote the buffer.","If the buffer is definitively corrupted, stop Vector, archive then delete the buffer files under `data_dir` (accepting data loss of buffered events), and restart — the buffer is rebuilt fresh.","While investigating, switch the buffer to `type = \"memory\"` so the pipeline can run without the failing disk buffer."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// preflight the data dir before starting the topology with a disk buffer\nlet meta = std::fs::metadata(&data_dir)?;\nif meta.is_dir() {\n    let _probe = std::fs::File::create(data_dir.join(\".write_probe\"))?\n        .and_then(|mut f| { use std::io::Write; f.write_all(b\"ok\") })?;\n    std::fs::remove_file(data_dir.join(\".write_probe\"))?;\n}\nlet available = fs2::available_space(&data_dir)?;\nassert!(available > min_required_bytes);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Monitor BufferReadError / buffer-related internal events — they precede the unrecoverable panic.","Alert on disk space for the volume backing data_dir and keep headroom under sustained backpressure.","Run one Vector version per data_dir; follow release notes when upgrading across disk-buffer format changes.","Back up or archive data_dir before experimental upgrades; be prepared to drop corrupted buffer files (accepting loss).","Consider buffers.type = \"memory\" for pipelines where losing the disk buffer is unacceptable operationally."],"tags":["rust","vector","disk-buffer","backpressure","corruption","panic"],"backgroundTag":"disk-buffer-corruption","analyzedSha":"3708c39b12a93212ed8b8d7510b4cc7769cb5864","analyzedAt":"2026-08-20T07:02:18.786Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T20:17:18.057Z"}