{"record":{"id":"8d237f2f1f9eeb72","repo":"quickwit-oss/quickwit","slug":"lock-should-not-be-poisoned-8d237f","errorCode":null,"errorMessage":"lock should not be poisoned","messagePattern":"lock should not be poisoned","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-common/src/tower/pool.rs","lineNumber":93,"sourceCode":"                    match change {\n                        Change::Insert(key, service) => {\n                            pool.insert(key, service);\n                        }\n                        Change::Remove(key) => {\n                            pool.remove(&key);\n                        }\n                    }\n                })\n                .await;\n        };\n        tokio::spawn(future);\n    }\n\n    /// Returns whether the pool is empty.\n    pub fn is_empty(&self) -> bool {\n        self.pool\n            .read()\n            .expect(\"lock should not be poisoned\")\n            .is_empty()\n    }\n\n    /// Returns the number of values in the pool.\n    pub fn len(&self) -> usize {\n        self.pool.read().expect(\"lock should not be poisoned\").len()\n    }\n\n    /// Returns all the keys in the pool.\n    pub fn keys(&self) -> Vec<K> {\n        self.pool\n            .read()\n            .expect(\"lock should not be poisoned\")\n            .keys()\n            .cloned()\n            .collect()\n    }\n","sourceCodeStart":75,"sourceCodeEnd":111,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-common/src/tower/pool.rs#L75-L111","documentation":"This `.expect(\"lock should not be poisoned\")` panic occurs in `Pool::is_empty` (quickwit-common/src/tower/pool.rs) when the pool's internal `RwLock` is poisoned. A `std::sync::RwLock` becomes poisoned when a thread panics while holding the lock (here, the write guard used when updating the pool, e.g. inserting/removing channel instances). Once poisoned, every subsequent lock acquisition with `.expect` panics, turning an earlier failure in another thread into repeated panics across the pool API.","triggerScenarios":"Another thread panicked while holding the pool's write lock (e.g. inside `insert`/`remove`/pool-update code), and afterwards any call to `is_empty` (or `len`, or channel acquisition) unwraps the poisoned guard and panics.","commonSituations":"A panic inside a closure that mutates the pool while iterating its members (e.g. a failing p2p/gRPC channel constructor), leaving the lock poisoned; cascading failures where the first panic is misattributed to `is_empty` in logs.","solutions":["Find and fix the original panic that occurred while a thread held the pool's write lock — the poisoned-lock panic is only a symptom.","On the caller side, catch the panic (or inspect results via `std::panic::catch_unwind`) when the pool may be shared with code that can panic.","If you control the code, switch to poisoning-tolerant access (`read().unwrap_or_else(PoisonError::into_inner)`) to recover the pool state, though only after fixing the root cause.","Review pool mutation paths (insert/remove of members) for fallible operations that can panic under lock."],"exampleFix":"// before: fixing symptom only\nif pool.is_empty() { /* panics after a prior lock-holder panic */ }\n\n// after: locate the real panic source inside the mutation path\npool.insert(name, make_channel()?); // return Err instead of panicking while holding the lock\nif pool.is_empty() { /* safe once root cause is fixed */ }","handlingStrategy":"try-catch","validationCode":"// Cannot detect poisoning without touching the lock; probe defensively:\nfn pool_readable(pool: &Pool) -> bool {\n    // if this returns false the lock was poisoned by an earlier panic\n    std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| pool.is_empty())).is_ok()\n}","typeGuard":null,"tryCatchPattern":"let is_empty = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| pool.is_empty()))\n    .unwrap_or_else(|_| {\n        tracing::error!(\"pool lock poisoned by an earlier panic; recycling pool\");\n        true\n    });","preventionTips":["Keep code that runs while holding pool locks panic-free (return Results instead of panicking).","Treat a poisoned-lock panic as a symptom — hunt the original panic that held the write guard.","Consider poisoning-tolerant recovery (PoisonError::into_inner) only after fixing root causes.","Add panic hooks/logging to identify which mutation path panicked first."],"tags":["rust","concurrency","mutex-poisoning","panic","tower"],"backgroundTag":"mutex-poisoned","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}