{"record":{"id":"e0d4c0bd5205a8a4","repo":"quickwit-oss/quickwit","slug":"receiver-lives-longer-than-sender","errorCode":null,"errorMessage":"Receiver lives longer than sender","messagePattern":"Receiver lives longer than sender","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-search/src/search_permit_provider.rs","lineNumber":194,"sourceCode":"    ///\n    /// If `offload_threshold` is 0, all splits are offloaded.\n    /// If `offload_threshold` is usize::MAX, all splits are processed locally.\n    pub(crate) async fn get_permits_with_offload(\n        &self,\n        task_metadata: LeafSearchTaskMetadata,\n        offload_threshold: usize,\n    ) -> Vec<SearchPermitFuture> {\n        if task_metadata.splits.is_empty() {\n            return Vec::new();\n        }\n        let (permit_sender, permit_receiver) = oneshot::channel();\n        self.message_sender\n            .send(SearchPermitMessage::RequestWithOffload {\n                permit_resp_tx: permit_sender,\n                task_metadata,\n                offload_threshold,\n            })\n            .expect(\"Receiver lives longer than sender\");\n        permit_receiver\n            .await\n            .expect(\"Receiver lives longer than sender\")\n    }\n}\n\nstruct SearchPermitActor {\n    msg_receiver: mpsc::UnboundedReceiver<SearchPermitMessage>,\n    msg_sender: mpsc::WeakUnboundedSender<SearchPermitMessage>,\n    num_warmup_slots_available: usize,\n    /// Note it is possible for memory_allocated to exceed memory_budget temporarily,\n    /// if and only if a split leaf search task ended up using more than `initial_allocation`.\n    /// When it happens, new permits will not be assigned until the memory is freed.\n    total_memory_budget: u64,\n    total_memory_allocated: u64,\n    /// Sum of [`SplitSearchTaskMetadata::job_cost`] for all queued and active tasks.\n    ///\n    /// Incremented when a task enters [`Self::permits_requests`], decremented when","sourceCodeStart":176,"sourceCodeEnd":212,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-search/src/search_permit_provider.rs#L176-L212","documentation":"get_permits_with_offload sends a RequestWithOffload message over an unbounded mpsc channel to the SearchPermitActor and expects the send to succeed. The send only fails when the receiver is dropped, i.e. the permit actor has shut down while a search request is still trying to acquire permits; the expect message 'Receiver lives longer than sender' asserts the actor outlives in-flight requests. The first expect (line 194) covers the send itself.","triggerScenarios":"Calling get_permits (→ get_permits_with_offload) after the search permit actor (root search actor's permit provider) has been stopped/killed — e.g. actor supervision shut down the mailbox while a leaf/root search task still holds a WeakUnboundedSender and attempts to request permits.","commonSituations":"Node shutdown or actor restart (config reload, control-plane churn) concurrent with in-flight search requests; tests that drop the actor handle before awaiting permits; hot-reload paths that rebuild the search actor while old futures are still polling.","solutions":["Ensure the permit provider actor outlives all search requests: keep a strong sender reference for the actor's lifetime and only drop it after in-flight requests complete.","Track shutdown properly: cancel or drain outstanding permit requests when the actor stops instead of letting them send into a dead channel.","If you maintain this code, replace the expect with an explicit error (e.g. 'search permit provider is shut down') so callers fail gracefully."],"exampleFix":"// before\nself.message_sender\n    .send(SearchPermitMessage::RequestWithOffload { ... })\n    .expect(\"Receiver lives longer than sender\");\n// after\nself.message_sender\n    .send(SearchPermitMessage::RequestWithOffload { ... })\n    .map_err(|_| anyhow::anyhow!(\"search permit provider is no longer running\"))?;","handlingStrategy":"try-catch","validationCode":"// before requesting permits\nif provider_is_shut_down() { return Err(anyhow::anyhow!(\"permit provider unavailable\")); }","typeGuard":null,"tryCatchPattern":"match sender.send(msg) {\n    Ok(()) => { /* await response */ }\n    Err(_) => return Err(anyhow::anyhow!(\"search permit provider is no longer running\")),\n}","preventionTips":["Keep a strong sender to the permit actor for the lifetime of in-flight searches.","Cancel/drain pending permit requests during actor shutdown.","Watch actor supervision logs: actor restarts while searches are queued cause this panic."],"tags":["rust","actors","channel","shutdown","mpsc"],"backgroundTag":"channel-closed","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}