quickwit-oss/quickwit · error

Receiver should live longer than sender

Error message

Receiver should live longer than sender

What it means

send_if_still_running sends a message to the permit provider's event loop through a Weak sender. The expect asserts the receiver (the actor event loop) is still alive whenever a strong sender exists. If the channel is closed despite the sender being upgradable, the lifecycle assumption is broken and the code panics.

Solutions

1. Ensure the permit provider event loop outlives all SearchPermit holders (hold its mailbox in the service root)
2. Audit shutdown ordering: stop sources/requests before dropping the receiver
3. Replace expect with graceful handling of SendError when a deliberate shutdown is in progress

Example fix

// before
sender.send(msg).expect("Receiver should live longer than sender");
// after
if sender.send(msg).is_err() {
    debug!("permit provider receiver already closed; dropping message");
}

// Check liveness before sending:
if search_permit_provider_msg_sender.strong_count() == 0 { /* provider gone */ }

fn receiver_alive(sender: &Weak<UnboundedSender<SearchPermitMessage>>) -> bool {
    sender.upgrade().map(|s| !s.is_closed()).unwrap_or(false)
}

// Replace expect for shutdown tolerance:
if let Some(sender) = self.msg_sender.upgrade() {
    let _ = sender.send(msg); // swallow SendError during shutdown
}

When it happens

Trigger: update_memory_usage, free_warmup_slot, or SearchPermit::drop attempt to send a message after the event loop's receiver was dropped or its channel closed, while the sender is still upgradeable.

Common situations: Shutting down the search service while in-flight requests still hold SearchPermits; ordering issues during actor teardown where the receiver dies before all permit holders drop.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• 60 should be non-zero
• closure always returns Some
• compactor service enabled but no compaction client available
• `doc_batch` should not be empty
• Failed to serialize QueryAst

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/3106ae0f52153fbe. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-search/src/search_permit_provider.rs:484

    pub fn free_warmup_slot(&mut self) {
        if self.warmup_slot_freed {
            return;
        }
        self.warmup_slot_freed = true;
        self.send_if_still_running(SearchPermitMessage::FreeWarmupSlot);
    }

    pub fn memory_allocation(&self) -> ByteSize {
        ByteSize(self.memory_allocation)
    }

    fn send_if_still_running(&self, msg: SearchPermitMessage) {
        if let Some(sender) = self.msg_sender.upgrade() {
            sender
                .send(msg)
                // Receiver instance in the event loop is never dropped or
                // closed as long as there is a strong sender reference.
                .expect("Receiver should live longer than sender");
        }
    }
}

impl Drop for SearchPermit {
    fn drop(&mut self) {
        let prev = self
            .total_job_cost
            .fetch_update(Ordering::Relaxed, Ordering::Relaxed, |v| {
                Some(v.saturating_sub(self.job_cost))
            })
            .expect("closure always returns Some");
        if self.job_cost > prev {
            warn!(
                job_cost = self.job_cost,
                total_job_cost = prev,
                "job cost underflow: more job cost released than allocated"
            );

View on GitHub (pinned to a39730c5cd)