{"record":{"id":"a3245c987a9fd740","repo":"influxdata/influxdb","slug":"not-poisoned-a3245c","errorCode":null,"errorMessage":"not poisoned","messagePattern":"not poisoned","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"core/object_store_mem_cache/src/cache_system/s3_fifo_cache/mod.rs","lineNumber":411,"sourceCode":"#[derive(Debug, Default, Clone)]\nstruct S3FifoInstrument {\n    caches: Arc<Mutex<BTreeMap<&'static str, Weak<dyn S3FifoStatProvider>>>>,\n}\n\nimpl S3FifoInstrument {\n    const INSTRUMENT_NAME: &str = \"s3_fifo_instrument\";\n    const METRIC_NAME_ENTRIES: &str = \"s3_fifo_instrument_entries\";\n    const METRIC_NAME_TOMBSTONES: &str = \"s3_fifo_instrument_tombstones\";\n    const METRIC_NAME_BYTES: &str = \"s3_fifo_instrument_bytes\";\n\n    fn register_cache<K, V>(&self, name: &'static str, cache: &Arc<S3Fifo<K, V>>)\n    where\n        K: Debug + Eq + Hash + HasSize + Send + Sync + 'static + ?Sized,\n        V: HasSize + InUse + Send + Sync + 'static,\n    {\n        self.caches\n            .lock()\n            .expect(\"not poisoned\")\n            .entry(name)\n            .or_insert_with(|| Arc::downgrade(cache) as _);\n    }\n}\n\nimpl metric::Instrument for S3FifoInstrument {\n    fn report(&self, reporter: &mut dyn metric::Reporter) {\n        let stats = {\n            let caches = self.caches.lock().expect(\"not poisoned\");\n            caches\n                .iter()\n                .flat_map(|(name, cache)| {\n                    let cache = cache.upgrade()?;\n                    let stats = cache.statistics();\n                    Some((*name, stats))\n                })\n                .collect::<Vec<_>>()\n        };","sourceCodeStart":393,"sourceCodeEnd":429,"githubUrl":"https://github.com/influxdata/influxdb/blob/d28e26e048401c53cbb98cf2d6ab0cf1e98048ca/core/object_store_mem_cache/src/cache_system/s3_fifo_cache/mod.rs#L393-L429","documentation":"S3FifoInstrument::register_cache locks an instrument-internal Mutex<HashMap<..>> with .expect(\"not poisoned\"). std sync mutexes become poisoned when any thread panics while holding them, so this panic fires on the FIRST registration after an earlier panic occurred while that same lock was held (in register_cache or in S3FifoInstrument::report). The panic you see is always a secondary symptom: the root cause is the original panic under the lock.","triggerScenarios":"Calling S3FifoCache::new()/register() (which calls register_cache) after a previous panic happened while the caches mutex was held, e.g. a panic inside the flat_map in report() (statistics(), cache.upgrade(), or the metric::Reporter) or inside another concurrent registration.","commonSituations":"A metric reporter or V::statistics() implementation panicking during a periodic metrics scrape; any user code panicking inside instrumentation callbacks; upgrading the crate so a new panic path runs under the lock; running with panic=abort makes the first panic fatal instead.","solutions":["Search the logs for the FIRST panic before this one — it identifies the code that poisoned the lock; fix that code (or the data that made it panic).","Restart the process: a poisoned std Mutex never heals, so every later registration/report will panic until restart.","If you control the value types, ensure V: HasSize/InUse implementations and Reporter callbacks used under this lock cannot panic.","As a library-level hardening, replace .lock().expect(\"not poisoned\") with .lock().unwrap_or_else(|e| e.into_inner()) (or use parking_lot, whose mutexes are unpoisoned) so instrumentation survives a panicking report."],"exampleFix":"// before\nself.caches\n    .lock()\n    .expect(\"not poisoned\")\n    .entry(name)\n    .or_insert_with(|| Arc::downgrade(cache) as _);\n\n// after (recover from poisoning instead of panicking)\nself.caches\n    .lock()\n    .unwrap_or_else(|e| e.into_inner())\n    .entry(name)\n    .or_insert_with(|| Arc::downgrade(cache) as _);","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// guard cache registration / instrumentation setup so one poisoned\n// instrument cannot take down startup or the metrics loop\nlet instrument = std::panic::catch_unwind(|| {\n    S3FifoCache::<str, Bytes>::new(config, &registry)\n})\n.map_err(|p| classify_panic(&p))?; // e.g. downcast to &str/Pattern and match \"not poisoned\"","preventionTips":["Treat any panic inside metric reporters or statistics callbacks as P1 — they run under instrument locks and poison them.","Log first-panic context (panic payload + backtrace) so the poisoning origin is diagnosable when 'not poisoned' appears later.","If you fork the crate, prefer .lock().unwrap_or_else(|e| e.into_inner()) or parking_lot::Mutex for instrumentation paths.","Decide explicitly between panic=unwind (poisoning possible, recoverable) and panic=abort (first panic is fatal) per service."],"tags":["rust","mutex","poisoned-lock","s3-fifo-cache","metrics","panic"],"backgroundTag":"poisoned-lock","analyzedSha":"d28e26e048401c53cbb98cf2d6ab0cf1e98048ca","analyzedAt":"2026-08-16T19:53:34.623Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}