influxdata/influxdb · error

not poisoned

Error message

not poisoned

What it means

S3FifoInstrument::register_cache locks an instrument-internal Mutex<HashMap<..>> with .expect("not poisoned"). std sync mutexes become poisoned when any thread panics while holding them, so this panic fires on the FIRST registration after an earlier panic occurred while that same lock was held (in register_cache or in S3FifoInstrument::report). The panic you see is always a secondary symptom: the root cause is the original panic under the lock.

Source

Thrown at core/object_store_mem_cache/src/cache_system/s3_fifo_cache/mod.rs:411

#[derive(Debug, Default, Clone)]
struct S3FifoInstrument {
    caches: Arc<Mutex<BTreeMap<&'static str, Weak<dyn S3FifoStatProvider>>>>,
}

impl S3FifoInstrument {
    const INSTRUMENT_NAME: &str = "s3_fifo_instrument";
    const METRIC_NAME_ENTRIES: &str = "s3_fifo_instrument_entries";
    const METRIC_NAME_TOMBSTONES: &str = "s3_fifo_instrument_tombstones";
    const METRIC_NAME_BYTES: &str = "s3_fifo_instrument_bytes";

    fn register_cache<K, V>(&self, name: &'static str, cache: &Arc<S3Fifo<K, V>>)
    where
        K: Debug + Eq + Hash + HasSize + Send + Sync + 'static + ?Sized,
        V: HasSize + InUse + Send + Sync + 'static,
    {
        self.caches
            .lock()
            .expect("not poisoned")
            .entry(name)
            .or_insert_with(|| Arc::downgrade(cache) as _);
    }
}

impl metric::Instrument for S3FifoInstrument {
    fn report(&self, reporter: &mut dyn metric::Reporter) {
        let stats = {
            let caches = self.caches.lock().expect("not poisoned");
            caches
                .iter()
                .flat_map(|(name, cache)| {
                    let cache = cache.upgrade()?;
                    let stats = cache.statistics();
                    Some((*name, stats))
                })
                .collect::<Vec<_>>()
        };

View on GitHub (pinned to d28e26e048)

Solutions

1. Search the logs for the FIRST panic before this one — it identifies the code that poisoned the lock; fix that code (or the data that made it panic).
2. Restart the process: a poisoned std Mutex never heals, so every later registration/report will panic until restart.
3. If you control the value types, ensure V: HasSize/InUse implementations and Reporter callbacks used under this lock cannot panic.
4. As a library-level hardening, replace .lock().expect("not poisoned") with .lock().unwrap_or_else(|e| e.into_inner()) (or use parking_lot, whose mutexes are unpoisoned) so instrumentation survives a panicking report.

Example fix

// before
self.caches
    .lock()
    .expect("not poisoned")
    .entry(name)
    .or_insert_with(|| Arc::downgrade(cache) as _);

// after (recover from poisoning instead of panicking)
self.caches
    .lock()
    .unwrap_or_else(|e| e.into_inner())
    .entry(name)
    .or_insert_with(|| Arc::downgrade(cache) as _);

// guard cache registration / instrumentation setup so one poisoned
// instrument cannot take down startup or the metrics loop
let instrument = std::panic::catch_unwind(|| {
    S3FifoCache::<str, Bytes>::new(config, &registry)
})
.map_err(|p| classify_panic(&p))?; // e.g. downcast to &str/Pattern and match "not poisoned"

When it happens

Trigger: Calling S3FifoCache::new()/register() (which calls register_cache) after a previous panic happened while the caches mutex was held, e.g. a panic inside the flat_map in report() (statistics(), cache.upgrade(), or the metric::Reporter) or inside another concurrent registration.

Common situations: A metric reporter or V::statistics() implementation panicking during a periodic metrics scrape; any user code panicking inside instrumentation callbacks; upgrading the crate so a new panic path runs under the lock; running with panic=abort makes the first panic fatal instead.

Related errors

• not poisoned
• ghost queue is NOT empty
• cannot fit duration into u64
• metric should be in progress
• no metric in progress

AI-assisted analysis of influxdata/influxdb@d28e26e048 (2026-08-16). Data as JSON: /api/errors/a3245c987a9fd740. Report an issue: GitHub.