{"record":{"id":"a8511699734125de","repo":"vectordotdev/vector","slug":"time-ewma-gauge-mutex-poisoned","errorCode":null,"errorMessage":"time ewma gauge mutex poisoned","messagePattern":"time ewma gauge mutex poisoned","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"lib/vector-common/src/stats/ewma_gauge.rs","lineNumber":54,"sourceCode":"pub struct TimeEwmaGauge {\n    gauge: Gauge,\n    ewma: Arc<Mutex<TimeEwma>>,\n}\n\nimpl TimeEwmaGauge {\n    #[must_use]\n    pub fn new(gauge: Gauge, half_life_seconds: f64) -> Self {\n        let ewma = Arc::new(Mutex::new(TimeEwma::new(half_life_seconds)));\n        Self { gauge, ewma }\n    }\n\n    /// Records a new value, updates the EWMA, and sets the gauge accordingly.\n    ///\n    /// # Panics\n    ///\n    /// Panics if the EWMA mutex is poisoned.\n    pub fn record(&self, value: f64, reference: Instant) {\n        let mut ewma = self.ewma.lock().expect(\"time ewma gauge mutex poisoned\");\n        let average = ewma.update(value, reference);\n        self.gauge.set(average);\n    }\n}\n","sourceCodeStart":36,"sourceCodeEnd":59,"githubUrl":"https://github.com/vectordotdev/vector/blob/3708c39b12a93212ed8b8d7510b4cc7769cb5864/lib/vector-common/src/stats/ewma_gauge.rs#L36-L59","documentation":"TimeEwmaGauge (lib/vector-common/src/stats/ewma_gauge.rs) guards its TimeEwma state with a std::sync::Mutex, and record() calls lock().expect(\"time ewma gauge mutex poisoned\"). A Rust mutex becomes poisoned when a thread panics while holding it; every later lock() then returns Err(PoisonError), which this expect turns into a panic. The critical section only runs ewma.update() and gauge.set(), so this panic means another panic already happened inside record() and the process is now failing on every subsequent metric update.","triggerScenarios":"Calling TimeEwmaGauge::record(value, reference) after an earlier call to record() on a clone of the gauge panicked between acquiring and releasing the mutex (inside TimeEwma::update or Gauge::set). Any clone shares the same Arc<Mutex<TimeEwma>>, so a panic in one topology component poisons the gauge for all users of it.","commonSituations":"Almost never seen in healthy Vector deployments; it surfaces when the process is already tearing down after an unrelated panic (e.g. a metrics-recording code path that panicked on a NaN/overflow), or in embedding code that clones the gauge across panicking tasks. Repeated 'time ewma gauge mutex poisoned' lines are a symptom, not the root cause.","solutions":["Search the logs back to the FIRST panic in the process — that panic poisoned the mutex; fix or report that root cause rather than this message","If you embed Vector and construct TimeEwmaGauge yourself, recover from poisoning with self.ewma.lock().unwrap_or_else(|e| e.into_inner()) since the EWMA state is not memory-unsafe to keep using","Restart the Vector process to clear the poisoned lock if the root-cause panic was a one-off (OOM-killed threads, transient resource exhaustion)","If the underlying panic is reproducible, capture a backtrace (RUST_BACKTRACE=1) and open an issue in vectordot/vector with the triggering configuration"],"exampleFix":"// before\nlet mut ewma = self.ewma.lock().expect(\"time ewma gauge mutex poisoned\");\nlet average = ewma.update(value, reference);\nself.gauge.set(average);\n\n// after (poisoning-tolerant: EWMA state is plain f64s, safe to keep using)\nlet mut ewma = self.ewma.lock().unwrap_or_else(|poisoned| poisoned.into_inner());\nlet average = ewma.update(value, reference);\nself.gauge.set(average);","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"// If you own the call site, tolerate poisoning instead of expect():\n// the TimeEwma state is plain f64s and safe to keep using.\nmatch self.ewma.lock() {\n    Ok(guard) => { let avg = guard.update(value, reference); self.gauge.set(avg); }\n    Err(poisoned) => { let mut guard = poisoned.into_inner(); let avg = guard.update(value, reference); self.gauge.set(avg); }\n}","preventionTips":["Treat the first panic in the process as the real incident; a poisoned-mutex message is always downstream of an earlier panic","Catch panics at component boundaries (std::panic::catch_unwind around tasks) so one panicking recorder cannot poison shared gauges","Keep metric-recording code arithmetic-simple so the lock's critical section cannot panic"],"tags":["rust","mutex","poisoning","metrics","ewma","panic"],"backgroundTag":"mutex-poisoned","analyzedSha":"3708c39b12a93212ed8b8d7510b4cc7769cb5864","analyzedAt":"2026-08-20T07:02:18.786Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}