influxdata/influxdb · error

not poisoned

Error message

not poisoned

What it means

The Mailbox in object_store_mem_cache's cache hooks coordinates task wakers using a Mutex<Vec<Waker>> plus an AtomicUsize counter. notify() locks the waker list and wakes every registered waker while holding the lock; if that code panics (a waker.wake() implementation panicking is the classic cause), the mutex is poisoned and every later lock() in notify()/poll_next() dies on this expect. The 'not poisoned' message is a downstream casualty - the real failure is the earlier panic in your logs.

Source

Thrown at core/object_store_mem_cache/src/cache_system/hook/notify/mod.rs:24

    },
    task::{Context, Poll, Waker},
};

use futures::Stream;

#[cfg(test)]
pub(crate) mod test_utils;

#[derive(Debug, Default)]
pub(crate) struct Mailbox {
    counter: AtomicUsize,
    wakers: Mutex<Vec<Waker>>,
}

impl Mailbox {
    /// Notify all notifiers
    pub(crate) fn notify(&self) {
        let mut guard = self.wakers.lock().expect("not poisoned");

        // bump counter AFTER acquiring lock but before notifying wakers
        self.counter.fetch_add(1, Ordering::SeqCst);

        for waker in guard.drain(..) {
            waker.wake();
        }
    }

    /// Notifier.
    pub(crate) fn notifier(self: &Arc<Self>) -> Notifier {
        Notifier {
            mailbox: Arc::downgrade(self),
            counter: 0,
        }
    }
}

View on GitHub (pinned to d28e26e048)

Solutions

1. Search the logs for the FIRST panic/backtrace - this expect is only a symptom; fix the original panic
2. Never panic inside wake()/poll implementations registered with this cache (validate waker invariants before use)
3. Patch defensive recovery: lock().unwrap_or_else(|e| e.into_inner()) - the Vec<Waker> is plain data and safe to keep using
4. Stress-test the memory-cache layer (high concurrency get/put) to reproduce the original race

Example fix

// before
let mut guard = self.wakers.lock().expect("not poisoned");

// after: recover from poisoning instead of cascading the panic
let mut guard = self.wakers.lock().unwrap_or_else(|e| e.into_inner());

// if you maintain this code: recover instead of cascading
let mut guard = self.wakers.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
// Vec<Waker> is plain data; continuing is safe

When it happens

Trigger: A registered Waker panics inside wake() while notify() holds the lock (guard is alive across the drain/wake loop); subsequently every call to notify() panics here because the mutex is already poisoned.

Common situations: Custom executors or futures with panicking wakers/poll implementations; any panic while holding the mailbox lock in the cache layer; cascading failures after an initial panic elsewhere in the cache hook.

Related errors

• can only accept once
• never fails
• cannot fit duration into u64
• metric should be in progress
• no metric in progress

AI-assisted analysis of influxdata/influxdb@d28e26e048 (2026-08-16). Data as JSON: /api/errors/9d3ca40cab830ef7. Report an issue: GitHub.