quickwit-oss/quickwit · error

lock should not be poisoned

Error message

lock should not be poisoned

What it means

This `.expect("lock should not be poisoned")` panic occurs in `Pool::is_empty` (quickwit-common/src/tower/pool.rs) when the pool's internal `RwLock` is poisoned. A `std::sync::RwLock` becomes poisoned when a thread panics while holding the lock (here, the write guard used when updating the pool, e.g. inserting/removing channel instances). Once poisoned, every subsequent lock acquisition with `.expect` panics, turning an earlier failure in another thread into repeated panics across the pool API.

Source

Thrown at quickwit/quickwit-common/src/tower/pool.rs:93

                    match change {
                        Change::Insert(key, service) => {
                            pool.insert(key, service);
                        }
                        Change::Remove(key) => {
                            pool.remove(&key);
                        }
                    }
                })
                .await;
        };
        tokio::spawn(future);
    }

    /// Returns whether the pool is empty.
    pub fn is_empty(&self) -> bool {
        self.pool
            .read()
            .expect("lock should not be poisoned")
            .is_empty()
    }

    /// Returns the number of values in the pool.
    pub fn len(&self) -> usize {
        self.pool.read().expect("lock should not be poisoned").len()
    }

    /// Returns all the keys in the pool.
    pub fn keys(&self) -> Vec<K> {
        self.pool
            .read()
            .expect("lock should not be poisoned")
            .keys()
            .cloned()
            .collect()
    }

View on GitHub (pinned to a39730c5cd)

Solutions

1. Find and fix the original panic that occurred while a thread held the pool's write lock — the poisoned-lock panic is only a symptom.
2. On the caller side, catch the panic (or inspect results via `std::panic::catch_unwind`) when the pool may be shared with code that can panic.
3. If you control the code, switch to poisoning-tolerant access (`read().unwrap_or_else(PoisonError::into_inner)`) to recover the pool state, though only after fixing the root cause.
4. Review pool mutation paths (insert/remove of members) for fallible operations that can panic under lock.

Example fix

// before: fixing symptom only
if pool.is_empty() { /* panics after a prior lock-holder panic */ }

// after: locate the real panic source inside the mutation path
pool.insert(name, make_channel()?); // return Err instead of panicking while holding the lock
if pool.is_empty() { /* safe once root cause is fixed */ }

// Cannot detect poisoning without touching the lock; probe defensively:
fn pool_readable(pool: &Pool) -> bool {
    // if this returns false the lock was poisoned by an earlier panic
    std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| pool.is_empty())).is_ok()
}

let is_empty = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| pool.is_empty()))
    .unwrap_or_else(|_| {
        tracing::error!("pool lock poisoned by an earlier panic; recycling pool");
        true
    });

When it happens

Trigger: Another thread panicked while holding the pool's write lock (e.g. inside `insert`/`remove`/pool-update code), and afterwards any call to `is_empty` (or `len`, or channel acquisition) unwraps the poisoned guard and panics.

Common situations: A panic inside a closure that mutates the pool while iterating its members (e.g. a failing p2p/gRPC channel constructor), leaving the lock poisoned; cascading failures where the first panic is misattributed to `is_empty` in logs.

Related errors

• lock should not be poisoned
• request should be set
• `poll_ready` should be called before `call`
• closure always returns Some
• node not found in pending

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/8d237f2f1f9eeb72. Report an issue: GitHub.