vectordotdev/vector · error

time ewma gauge mutex poisoned

Error message

time ewma gauge mutex poisoned

What it means

TimeEwmaGauge (lib/vector-common/src/stats/ewma_gauge.rs) guards its TimeEwma state with a std::sync::Mutex, and record() calls lock().expect("time ewma gauge mutex poisoned"). A Rust mutex becomes poisoned when a thread panics while holding it; every later lock() then returns Err(PoisonError), which this expect turns into a panic. The critical section only runs ewma.update() and gauge.set(), so this panic means another panic already happened inside record() and the process is now failing on every subsequent metric update.

Source

Thrown at lib/vector-common/src/stats/ewma_gauge.rs:54

pub struct TimeEwmaGauge {
    gauge: Gauge,
    ewma: Arc<Mutex<TimeEwma>>,
}

impl TimeEwmaGauge {
    #[must_use]
    pub fn new(gauge: Gauge, half_life_seconds: f64) -> Self {
        let ewma = Arc::new(Mutex::new(TimeEwma::new(half_life_seconds)));
        Self { gauge, ewma }
    }

    /// Records a new value, updates the EWMA, and sets the gauge accordingly.
    ///
    /// # Panics
    ///
    /// Panics if the EWMA mutex is poisoned.
    pub fn record(&self, value: f64, reference: Instant) {
        let mut ewma = self.ewma.lock().expect("time ewma gauge mutex poisoned");
        let average = ewma.update(value, reference);
        self.gauge.set(average);
    }
}

View on GitHub (pinned to 3708c39b12)

Solutions

  1. Search the logs back to the FIRST panic in the process — that panic poisoned the mutex; fix or report that root cause rather than this message
  2. If you embed Vector and construct TimeEwmaGauge yourself, recover from poisoning with self.ewma.lock().unwrap_or_else(|e| e.into_inner()) since the EWMA state is not memory-unsafe to keep using
  3. Restart the Vector process to clear the poisoned lock if the root-cause panic was a one-off (OOM-killed threads, transient resource exhaustion)
  4. If the underlying panic is reproducible, capture a backtrace (RUST_BACKTRACE=1) and open an issue in vectordot/vector with the triggering configuration

Example fix

// before
let mut ewma = self.ewma.lock().expect("time ewma gauge mutex poisoned");
let average = ewma.update(value, reference);
self.gauge.set(average);

// after (poisoning-tolerant: EWMA state is plain f64s, safe to keep using)
let mut ewma = self.ewma.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let average = ewma.update(value, reference);
self.gauge.set(average);
Defensive patterns

Strategy: fallback

Try / catch

// If you own the call site, tolerate poisoning instead of expect():
// the TimeEwma state is plain f64s and safe to keep using.
match self.ewma.lock() {
    Ok(guard) => { let avg = guard.update(value, reference); self.gauge.set(avg); }
    Err(poisoned) => { let mut guard = poisoned.into_inner(); let avg = guard.update(value, reference); self.gauge.set(avg); }
}

Prevention

When it happens

Trigger: Calling TimeEwmaGauge::record(value, reference) after an earlier call to record() on a clone of the gauge panicked between acquiring and releasing the mutex (inside TimeEwma::update or Gauge::set). Any clone shares the same Arc<Mutex<TimeEwma>>, so a panic in one topology component poisons the gauge for all users of it.

Common situations: Almost never seen in healthy Vector deployments; it surfaces when the process is already tearing down after an unrelated panic (e.g. a metrics-recording code path that panicked on a NaN/overflow), or in embedding code that clones the gauge across panicking tasks. Repeated 'time ewma gauge mutex poisoned' lines are a symptom, not the root cause.

Related errors

AI-assisted analysis of vectordotdev/vector@3708c39b12 (2026-08-20). Data as JSON: /api/errors/a8511699734125de. Report an issue: GitHub.