{"record":{"id":"18040d216a4c0526","repo":"vectordotdev/vector","slug":"file-server-exited-with-an-error-18040d","errorCode":null,"errorMessage":"file server exited with an error","messagePattern":"file server exited with an error","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/sources/kubernetes_logs/util.rs","lineNumber":45,"sourceCode":"    C: Sink<Vec<Line>> + Unpin + Send + 'static,\n    <C as Sink<Vec<Line>>>::Error: Error + Send,\n    S: Future + Unpin + Send + 'static,\n    <S as Future>::Output: Clone + Send + Sync,\n    <<PP as PathsProvider>::IntoIter as IntoIterator>::IntoIter: Send,\n{\n    let span = info_span!(\"file_server\");\n\n    // spawn_blocking shouldn't be needed: https://github.com/vectordotdev/vector/issues/23743\n    let join_handle = tokio::task::spawn_blocking(move || {\n        // These will need to be separated when this source is updated\n        // to support end-to-end acknowledgements.\n        let shutdown = shutdown.shared();\n        let shutdown2 = shutdown.clone();\n        let _enter = span.enter();\n\n        let rt = tokio::runtime::Handle::current();\n        let result = rt.block_on(file_server.run(chans, shutdown, shutdown2, checkpointer));\n        result.expect(\"file server exited with an error\")\n    });\n    join_handle.await\n}\n\npub async fn complete_with_deadline_on_signal<F, S>(\n    future: F,\n    signal: S,\n    deadline: Duration,\n) -> Result<<F as Future>::Output, tokio::time::error::Elapsed>\nwhere\n    F: Future,\n    S: Future<Output = ()>,\n{\n    pin_mut!(future);\n    pin_mut!(signal);\n    let future = match select(future, signal).await {\n        Either::Left((future_output, _)) => return Ok(future_output),\n        Either::Right(((), future)) => future,","sourceCodeStart":27,"sourceCodeEnd":63,"githubUrl":"https://github.com/vectordotdev/vector/blob/3708c39b12a93212ed8b8d7510b4cc7769cb5864/src/sources/kubernetes_logs/util.rs#L27-L63","documentation":"`file_server.run(...)` returns a Result; this `expect(\"file server exited with an error\")` panics the kubernetes_logs source's blocking task when the embedded file-source server exits with an error. That server tails kubelet log files, applies glob filtering, and maintains checkpoints, so failures such as filesystem-watcher (inotify) errors, checkpoint read/write errors on data_dir, or glob failures surface here and abort the whole kubernetes_logs source.","triggerScenarios":"kubernetes_logs running while inotify watches/instances are exhausted; data_dir read-only or full; checkpoint files under data_dir/k8spathscache missing permissions or corrupted; an internal glob failure while enumerating pod log paths.","commonSituations":"Nodes with very high pod churn exhausting `fs.inotify.max_user_watches`/`max_user_instances`; Vector's data_dir on a full or read-only volume; a stale checkpoint file owned by another user after running Vector once as root.","solutions":["Read the error printed just before the panic — the file_server error names the failing subsystem (watch, checkpoint IO, or glob)","Raise inotify limits on the node (`sysctl -w fs.inotify.max_user_watches=1048576`, likewise max_user_instances)","Repair data_dir: ensure write permission and free space; remove stale checkpoint files after stopping Vector","Upgrade Vector if the cause is an internal glob bug"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"# node pre-flight for the common causes\nsysctl fs.inotify.max_user_watches fs.inotify.max_user_instances\ndf -h /var/lib/vector          # data_dir free space\ntouch /var/lib/vector/.writetest && rm /var/lib/vector/.writetest","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Provision inotify limits (watches and instances) for expected pod/container churn","Keep Vector data_dir on a writable volume with disk-space alerts","Run kubernetes_logs under a supervisor (systemd Restart=on-failure) so transient fs errors recover by restart"],"tags":["kubernetes-logs","file-source","inotify","checkpoint","panic"],"backgroundTag":"file-watcher-error","analyzedSha":"3708c39b12a93212ed8b8d7510b4cc7769cb5864","analyzedAt":"2026-08-20T07:02:18.786Z","contentChangedAt":"2026-08-20T07:02:18.786Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}