{"record":{"id":"cdfcd290d433d013","repo":"vectordotdev/vector","slug":"failed-to-acquire-read-lock-on-recency-map","errorCode":null,"errorMessage":"Failed to acquire read lock on recency map","messagePattern":"Failed to acquire read lock on recency map","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"lib/vector-core/src/metrics/recorder.rs","lineNumber":64,"sourceCode":"        } else {\n            Some(Recency::new(\n                Clock::new(),\n                MetricKindMask::ALL,\n                global_timeout,\n                expire_metrics_per_metric_set,\n            ))\n        };\n        *(self.recency.write()).expect(\"Failed to acquire write lock on recency map\") = recency;\n    }\n\n    pub(super) fn visit_metrics(&self) -> Vec<Metric> {\n        let timestamp = Utc::now();\n\n        let mut metrics = Vec::new();\n        let recency = self\n            .recency\n            .read()\n            .expect(\"Failed to acquire read lock on recency map\");\n        let recency = recency.as_ref();\n\n        for (key, counter) in self.registry.get_counter_handles() {\n            if recency\n                .is_none_or(|recency| recency.should_store_counter(&key, &counter, &self.registry))\n            {\n                // NOTE this will truncate if the value is greater than 2**52.\n                #[allow(clippy::cast_precision_loss)]\n                let value = counter.get_inner().load(Ordering::Relaxed) as f64;\n                let value = MetricValue::Counter { value };\n                metrics.push(Metric::from_metric_kv(&key, value, timestamp));\n            }\n        }\n        for (key, gauge) in self.registry.get_gauge_handles() {\n            if recency\n                .is_none_or(|recency| recency.should_store_gauge(&key, &gauge, &self.registry))\n            {\n                let value = gauge.get_inner().load(Ordering::Relaxed);","sourceCodeStart":46,"sourceCodeEnd":82,"githubUrl":"https://github.com/vectordotdev/vector/blob/bdb87aeaa4c4ff27c0ba643c1c77b21bf2ef4013/lib/vector-core/src/metrics/recorder.rs#L46-L82","documentation":"Panic raised by `.expect()` on `self.recency.read()` inside `visit_metrics`. A std `RwLock` read acquisition only fails when the lock is poisoned — a writer panicked while holding or awaiting the lock — so this error means an earlier panic corrupted the shared recency-map state.","triggerScenarios":"Calling `visit_metrics` (the periodic metric-flush/visit path) after any thread panicked while holding the recency map's write lock, e.g. the panic from `set_expiry`'s own expect or a panic during recency-map mutation.","commonSituations":"Vector's internal metrics scrape loop calling `visit_metrics` on every flush interval after a one-time panic poisoned the map; every subsequent scrape then panics, typically cascading into task shutdown.","solutions":["Locate the first panic in the logs that poisoned the lock; that root cause must be fixed, not this symptom.","Replace std `RwLock` with `parking_lot::RwLock` (unpoisoned by design) or handle `PoisonError` via `into_inner()`.","Restart the process — poisoned locks never recover in-process.","Ensure metric-expiration config (`expire_metrics`, timeouts) does not trigger the upstream panic path."],"exampleFix":"// before\nlet recency = self\n    .recency\n    .read()\n    .expect(\"Failed to acquire read lock on recency map\");\n// after\nlet recency = self\n    .recency\n    .read()\n    .unwrap_or_else(|poisoned| poisoned.into_inner());","handlingStrategy":"try-catch","validationCode":"// Detect poisoning before the read\nlet usable = matches!(self.recency.try_read(), Ok(_) | Err(std::sync::TryLockError::WouldBlock));","typeGuard":"fn recency_readable<T>(lock: &std::sync::RwLock<T>) -> bool {\n    !matches!(lock.try_read(), Err(std::sync::TryLockError::Poisoned(_)))\n}","tryCatchPattern":"// Contain the visit in a panic barrier and recover gracefully\nlet metrics = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {\n    recorder.visit_metrics()\n}))\n.unwrap_or_default(); // return empty vec instead of crashing the scrape loop","preventionTips":["Fix any writer-side panic (e.g. in set_expiry) first; readers panic only after a writer poisoned the lock.","Use unwrap_or_else(|p| p.into_inner()) to tolerate poisoning in read-only scrape paths.","Avoid .expect() on lock acquisition in library code that runs on hot periodic paths.","Add a health check that surfaces a poisoned metrics lock before data pipelines crash."],"tags":["rust","locking","panic","poisoned-mutex"],"backgroundTag":"mutex-lock-poisoned","analyzedSha":"bdb87aeaa4c4ff27c0ba643c1c77b21bf2ef4013","analyzedAt":"2026-09-16T02:53:35.741Z","contentChangedAt":"2026-09-16T02:53:35.741Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}