What causes "mutex poisoned" in vectordotdev/vector?

Locate the first panic before the "mutex poisoned" messages in the logs and fix/upgrade for it

mutex poisoned

Error message

mutex poisoned

What it means

The utilization (busy/idle ratio) subsystem keeps per-component timers in a `std::sync::Mutex<HashMap<ComponentKey, Timer>>`. `add_component` locks it with `.expect("mutex poisoned")`; a panic anywhere inside a critical section on this mutex (insertion, removal, message handling) poisons it, after which every later `add_component` — each new component at startup or config reload — panics with the same message.

Source

Thrown at src/utilization.rs:229

/// Registry for components sending utilization data.
///
/// Cloning this is cheap and does not clone the underlying data.
#[derive(Clone)]
pub struct UtilizationRegistry {
    timers: Arc<Mutex<HashMap<ComponentKey, Timer>>>,
    timer_tx: Sender<UtilizationTimerMessage>,
}

impl UtilizationRegistry {
    /// Adds a new component to this utilization metric emitter
    ///
    /// Returns a sender which can be used to send utilization information back to the emitter
    pub(crate) fn add_component(
        &self,
        key: ComponentKey,
        gauge: Gauge,
    ) -> UtilizationComponentSender {
        self.timers.lock().expect("mutex poisoned").insert(
            key.clone(),
            Timer::new(
                gauge,
                #[cfg(debug_assertions)]
                key.id().into(),
            ),
        );
        UtilizationComponentSender {
            timer_tx: self.timer_tx.clone(),
            component_key: key,
        }
    }

    /// Removes a component from this utilization metric emitter
    pub(crate) fn remove_component(&self, key: &ComponentKey) {
        self.timers.lock().expect("mutex poisoned").remove(key);
    }
}

View on GitHub (pinned to 3708c39b12)

Solutions

Locate the first panic before the "mutex poisoned" messages in the logs and fix/upgrade for it
Restart Vector to clear the poisoned mutex
If embedding: recover with `lock().unwrap_or_else(|e| e.into_inner())` — the map is still usable for metrics — or use parking_lot::Mutex which cannot poison

Example fix

// before
self.timers.lock().expect("mutex poisoned").insert(key.clone(), Timer::new(gauge));
// after
self.timers
    .lock()
    .unwrap_or_else(|poisoned| poisoned.into_inner())
    .insert(key.clone(), Timer::new(gauge));

Defensive patterns

Strategy: fallback

Validate before calling

# shell: utilization poisoning always follows an earlier panic
journalctl -u vector | grep -m1 -B5 'panicked'

Try / catch

// Rust (embedding): best-effort metrics should not poison-shutdown the registry
let timers = self
    .timers
    .lock()
    .unwrap_or_else(|poisoned| poisoned.into_inner());

Prevention

Alert on the first panic, not the cascade of "mutex poisoned" messages
Restart to clear poisoned locks after fixing the original fault
For metrics-only shared state, prefer parking_lot or recovered locks over expect

When it happens

Trigger: A prior panic while the timers lock was held (during an earlier add_component/remove_component or a timer update in run_utilization); the next component registration then hits the poisoned lock.

Common situations: Appears cascading after an earlier panic during topology build or reload; the utilization metrics emitter then fails repeatedly for every new component, masking the original fault.

Related errors

AI-assisted analysis of vectordotdev/vector@3708c39b12 (2026-08-20). Data as JSON: /api/errors/c137ba5c9e3af493. Report an issue: GitHub.