quickwit-oss/quickwit · error

Receiver lives longer than sender

Error message

Receiver lives longer than sender

What it means

get_permits_with_offload sends a RequestWithOffload message over an unbounded mpsc channel to the SearchPermitActor and expects the send to succeed. The send only fails when the receiver is dropped, i.e. the permit actor has shut down while a search request is still trying to acquire permits; the expect message 'Receiver lives longer than sender' asserts the actor outlives in-flight requests. The first expect (line 194) covers the send itself.

Source

Thrown at quickwit/quickwit-search/src/search_permit_provider.rs:194

    ///
    /// If `offload_threshold` is 0, all splits are offloaded.
    /// If `offload_threshold` is usize::MAX, all splits are processed locally.
    pub(crate) async fn get_permits_with_offload(
        &self,
        task_metadata: LeafSearchTaskMetadata,
        offload_threshold: usize,
    ) -> Vec<SearchPermitFuture> {
        if task_metadata.splits.is_empty() {
            return Vec::new();
        }
        let (permit_sender, permit_receiver) = oneshot::channel();
        self.message_sender
            .send(SearchPermitMessage::RequestWithOffload {
                permit_resp_tx: permit_sender,
                task_metadata,
                offload_threshold,
            })
            .expect("Receiver lives longer than sender");
        permit_receiver
            .await
            .expect("Receiver lives longer than sender")
    }
}

struct SearchPermitActor {
    msg_receiver: mpsc::UnboundedReceiver<SearchPermitMessage>,
    msg_sender: mpsc::WeakUnboundedSender<SearchPermitMessage>,
    num_warmup_slots_available: usize,
    /// Note it is possible for memory_allocated to exceed memory_budget temporarily,
    /// if and only if a split leaf search task ended up using more than `initial_allocation`.
    /// When it happens, new permits will not be assigned until the memory is freed.
    total_memory_budget: u64,
    total_memory_allocated: u64,
    /// Sum of [`SplitSearchTaskMetadata::job_cost`] for all queued and active tasks.
    ///
    /// Incremented when a task enters [`Self::permits_requests`], decremented when

View on GitHub (pinned to a39730c5cd)

Solutions

1. Ensure the permit provider actor outlives all search requests: keep a strong sender reference for the actor's lifetime and only drop it after in-flight requests complete.
2. Track shutdown properly: cancel or drain outstanding permit requests when the actor stops instead of letting them send into a dead channel.
3. If you maintain this code, replace the expect with an explicit error (e.g. 'search permit provider is shut down') so callers fail gracefully.

Example fix

// before
self.message_sender
    .send(SearchPermitMessage::RequestWithOffload { ... })
    .expect("Receiver lives longer than sender");
// after
self.message_sender
    .send(SearchPermitMessage::RequestWithOffload { ... })
    .map_err(|_| anyhow::anyhow!("search permit provider is no longer running"))?;

// before requesting permits
if provider_is_shut_down() { return Err(anyhow::anyhow!("permit provider unavailable")); }

match sender.send(msg) {
    Ok(()) => { /* await response */ }
    Err(_) => return Err(anyhow::anyhow!("search permit provider is no longer running")),
}

When it happens

Trigger: Calling get_permits (→ get_permits_with_offload) after the search permit actor (root search actor's permit provider) has been stopped/killed — e.g. actor supervision shut down the mailbox while a leaf/root search task still holds a WeakUnboundedSender and attempts to request permits.

Common situations: Node shutdown or actor restart (config reload, control-plane churn) concurrent with in-flight search requests; tests that drop the actor handle before awaiting permits; hot-reload paths that rebuild the search actor while old futures are still polling.

Related errors

• Receiver should live longer than sender
• node not found in pending
• OTP logs or traces do not support VRL transforms
• position of a Kafka partition should never be EOF
• position of a Kinesis shard should never be EOF

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/e0d4c0bd5205a8a4. Report an issue: GitHub.