quickwit-oss/quickwit · error

Receiver should live longer than sender

Error message

Receiver should live longer than sender

What it means

send_if_still_running sends a message to the permit provider's event loop through a Weak sender. The expect asserts the receiver (the actor event loop) is still alive whenever a strong sender exists. If the channel is closed despite the sender being upgradable, the lifecycle assumption is broken and the code panics.

Source

Thrown at quickwit/quickwit-search/src/search_permit_provider.rs:484

    pub fn free_warmup_slot(&mut self) {
        if self.warmup_slot_freed {
            return;
        }
        self.warmup_slot_freed = true;
        self.send_if_still_running(SearchPermitMessage::FreeWarmupSlot);
    }

    pub fn memory_allocation(&self) -> ByteSize {
        ByteSize(self.memory_allocation)
    }

    fn send_if_still_running(&self, msg: SearchPermitMessage) {
        if let Some(sender) = self.msg_sender.upgrade() {
            sender
                .send(msg)
                // Receiver instance in the event loop is never dropped or
                // closed as long as there is a strong sender reference.
                .expect("Receiver should live longer than sender");
        }
    }
}

impl Drop for SearchPermit {
    fn drop(&mut self) {
        let prev = self
            .total_job_cost
            .fetch_update(Ordering::Relaxed, Ordering::Relaxed, |v| {
                Some(v.saturating_sub(self.job_cost))
            })
            .expect("closure always returns Some");
        if self.job_cost > prev {
            warn!(
                job_cost = self.job_cost,
                total_job_cost = prev,
                "job cost underflow: more job cost released than allocated"
            );

View on GitHub (pinned to a39730c5cd)

Solutions

1. Ensure the permit provider event loop outlives all SearchPermit holders (hold its mailbox in the service root)
2. Audit shutdown ordering: stop sources/requests before dropping the receiver
3. Replace expect with graceful handling of SendError when a deliberate shutdown is in progress

Example fix

// before
sender.send(msg).expect("Receiver should live longer than sender");
// after
if sender.send(msg).is_err() {
    debug!("permit provider receiver already closed; dropping message");
}

// Check liveness before sending:
if search_permit_provider_msg_sender.strong_count() == 0 { /* provider gone */ }

fn receiver_alive(sender: &Weak<UnboundedSender<SearchPermitMessage>>) -> bool {
    sender.upgrade().map(|s| !s.is_closed()).unwrap_or(false)
}

// Replace expect for shutdown tolerance:
if let Some(sender) = self.msg_sender.upgrade() {
    let _ = sender.send(msg); // swallow SendError during shutdown
}

When it happens

Trigger: update_memory_usage, free_warmup_slot, or SearchPermit::drop attempt to send a message after the event loop's receiver was dropped or its channel closed, while the sender is still upgradeable.

Common situations: Shutting down the search service while in-flight requests still hold SearchPermits; ordering issues during actor teardown where the receiver dies before all permit holders drop.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• node not found in pending
• OTP logs or traces do not support VRL transforms
• `doc_batch` should not be empty
• lock should not be poisoned
• request should be set

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/3106ae0f52153fbe. Report an issue: GitHub.