vectordotdev/vector · critical

Reader encountered unrecoverable error: {e:?}

Error message

Reader encountered unrecoverable error: {e:?}

What it means

The disk-v2 buffer reader inside `ReceiverAdapter::next` (lib/vector-buffers/src/topology/channel/receiver.rs:47-61) loops over `reader.next().await` and classifies each `Err`: recoverable errors are emitted as internal events and reading continues, but an error with no recoverable variant is considered fatal buffer corruption and panics, tearing down the topology task. This is intentional fail-fast behavior — the on-disk ledger/segment data is untrusted and cannot be safely skipped. Recoverable errors surfaced earlier (e.g. `BufferReadError` events) are the warning sign that disk reads are degraded.

Source

Thrown at lib/vector-buffers/src/topology/channel/receiver.rs:59

}

impl<T> ReceiverAdapter<T>
where
    T: Bufferable,
{
    pub(crate) async fn next(&mut self) -> Option<T> {
        match self {
            ReceiverAdapter::InMemory(rx) => rx.next().await,
            ReceiverAdapter::DiskV2(reader) => loop {
                match reader.next().await {
                    Ok(result) => break result,
                    Err(e) => match e.as_recoverable_error() {
                        Some(re) => {
                            // If we've hit a recoverable error, we'll emit an event to indicate as much but we'll still
                            // keep trying to read the next available record.
                            emit(re);
                        }
                        None => panic!("Reader encountered unrecoverable error: {e:?}"),
                    },
                }
            },
        }
    }
}

/// A buffer receiver.
///
/// The receiver handles retrieving events from the buffer, regardless of the overall buffer configuration.
///
/// If a buffer was configured to operate in "overflow" mode, then the receiver will be responsible
/// for querying the overflow buffer as well.  The ordering of events when operating in "overflow"
/// is undefined, as the receiver will try to manage polling both its own buffer, as well as the
/// overflow buffer, in order to fairly balance throughput.
#[derive(Debug)]
pub struct BufferReceiver<T: Bufferable> {
    base: ReceiverAdapter<T>,

View on GitHub (pinned to 3708c39b12)

Solutions

1. Check the obvious environmental causes first: free space on the `data_dir` volume, directory permissions, and disk/filesystem health (dmesg/SMART for I/O errors).
2. Restore version compatibility: restart with the same (or a format-compatible, per release notes) Vector version that wrote the buffer.
3. If the buffer is definitively corrupted, stop Vector, archive then delete the buffer files under `data_dir` (accepting data loss of buffered events), and restart — the buffer is rebuilt fresh.
4. While investigating, switch the buffer to `type = "memory"` so the pipeline can run without the failing disk buffer.

// preflight the data dir before starting the topology with a disk buffer
let meta = std::fs::metadata(&data_dir)?;
if meta.is_dir() {
    let _probe = std::fs::File::create(data_dir.join(".write_probe"))?
        .and_then(|mut f| { use std::io::Write; f.write_all(b"ok") })?;
    std::fs::remove_file(data_dir.join(".write_probe"))?;
}
let available = fs2::available_space(&data_dir)?;
assert!(available > min_required_bytes);

When it happens

Trigger: A Vector deployment configured with `buffers.type = "disk"` (disk v2, `data_dir` on persistent storage) where `BufferReader::next` returns a non-recoverable error: corrupted or truncated ledger/segment files after an unclean shutdown, unreadable data dir (permissions, failing disk, ENOSPC), or buffer files written by an incompatible Vector version.

Common situations: Hard power loss or OOM-kill while the disk buffer held data; running out of disk space under sustained backpressure; downgrading or skipping Vector versions across restarts so the on-disk format no longer parses; copying a `data_dir` between machines/versions; container deployments where the volume backing `data_dir` is remounted read-only.

Related errors

• effective reader file ID must be in the checkpoint window
• record was already validated
• event count should never exceed u64
• Event count for a record cannot exceed 2^64 events.
• a record with a next ID must have an event count

AI-assisted analysis of vectordotdev/vector@3708c39b12 (2026-08-20). Data as JSON: /api/errors/6a9d0ba77aa0f328. Report an issue: GitHub.