vectordotdev/vector · error

file server exited with an error

Error message

file server exited with an error

What it means

`file_server.run(...)` returns a Result; this `expect("file server exited with an error")` panics the kubernetes_logs source's blocking task when the embedded file-source server exits with an error. That server tails kubelet log files, applies glob filtering, and maintains checkpoints, so failures such as filesystem-watcher (inotify) errors, checkpoint read/write errors on data_dir, or glob failures surface here and abort the whole kubernetes_logs source.

Solutions

1. Read the error printed just before the panic — the file_server error names the failing subsystem (watch, checkpoint IO, or glob)
2. Raise inotify limits on the node (`sysctl -w fs.inotify.max_user_watches=1048576`, likewise max_user_instances)
3. Repair data_dir: ensure write permission and free space; remove stale checkpoint files after stopping Vector
4. Upgrade Vector if the cause is an internal glob bug

# node pre-flight for the common causes
sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances
df -h /var/lib/vector          # data_dir free space
touch /var/lib/vector/.writetest && rm /var/lib/vector/.writetest

When it happens

Trigger: kubernetes_logs running while inotify watches/instances are exhausted; data_dir read-only or full; checkpoint files under data_dir/k8spathscache missing permissions or corrupted; an internal glob failure while enumerating pod log paths.

Common situations: Nodes with very high pod churn exhausting `fs.inotify.max_user_watches`/`max_user_instances`; Vector's data_dir on a full or read-only volume; a stale checkpoint file owned by another user after running Vector once as root.

Related errors

• a record with a next ID must have an event count
• all json-file keys should be matched
• effective reader file ID must be in the checkpoint window
• Event count for a record cannot exceed 2^64 events.
• failed to read glob pattern

AI-assisted analysis of vectordotdev/vector@3708c39b12 (2026-08-20). Data as JSON: /api/errors/18040d216a4c0526. Report an issue: GitHub.

Appendix: source

Thrown at src/sources/kubernetes_logs/util.rs:45

    C: Sink<Vec<Line>> + Unpin + Send + 'static,
    <C as Sink<Vec<Line>>>::Error: Error + Send,
    S: Future + Unpin + Send + 'static,
    <S as Future>::Output: Clone + Send + Sync,
    <<PP as PathsProvider>::IntoIter as IntoIterator>::IntoIter: Send,
{
    let span = info_span!("file_server");

    // spawn_blocking shouldn't be needed: https://github.com/vectordotdev/vector/issues/23743
    let join_handle = tokio::task::spawn_blocking(move || {
        // These will need to be separated when this source is updated
        // to support end-to-end acknowledgements.
        let shutdown = shutdown.shared();
        let shutdown2 = shutdown.clone();
        let _enter = span.enter();

        let rt = tokio::runtime::Handle::current();
        let result = rt.block_on(file_server.run(chans, shutdown, shutdown2, checkpointer));
        result.expect("file server exited with an error")
    });
    join_handle.await
}

pub async fn complete_with_deadline_on_signal<F, S>(
    future: F,
    signal: S,
    deadline: Duration,
) -> Result<<F as Future>::Output, tokio::time::error::Elapsed>
where
    F: Future,
    S: Future<Output = ()>,
{
    pin_mut!(future);
    pin_mut!(signal);
    let future = match select(future, signal).await {
        Either::Left((future_output, _)) => return Ok(future_output),
        Either::Right(((), future)) => future,

View on GitHub (pinned to 3708c39b12)