quickwit-oss/quickwit · critical

More memory released than allocated, should never happen.

Error message

More memory released than allocated, should never happen.

What it means

This panic fires in the search permit provider's memory accounting when a release would drive total_memory_allocated below zero. It means the internal bookkeeping between allocation and release is broken: someone released memory that was never charged. The library treats this as an unrecoverable invariant violation and panics via expect.

Solutions

1. Check for double-release of the same permit (SearchPermit dropped more than once or release message sent twice)
2. Verify the memory allocation path actually increments total_memory_allocated for every release path that can fire
3. Reproduce with debug logging of alloc/release pairs to find the unpaired release
4. If hit after code changes, review recent changes to warmup slot freeing and assign_available_permits ordering

Example fix

// before
self.total_memory_allocated = self.total_memory_allocated.checked_sub(memory_size).expect("...");
// after
if let Some(new_total) = self.total_memory_allocated.checked_sub(memory_size) {
    self.total_memory_allocated = new_total;
} else {
    error!(memory_size, total = self.total_memory_allocated, "unpaired memory release");
    return; // or panic with diagnostic context
}

// Before releasing, assert the allocation was tracked:
assert!(memory_size <= provider.total_memory_allocated, "release exceeds allocated memory");

When it happens

Trigger: handle_message processes a memory release message whose memory_size was never added to total_memory_allocated, or the same allocation is released twice (e.g. duplicate drop/ack messages racing in the actor event loop).

Common situations: Actor message duplication or reordering bugs, custom sources or client code holding SearchPermits across actor restarts, or modifications to the permit provider that release warmup slots/memory out of order.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• 60 should be non-zero
• closure always returns Some
• compactor service enabled but no compaction client available
• `doc_batch` should not be empty
• Failed to serialize QueryAst

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/374dc47cd21f039f. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-search/src/search_permit_provider.rs:384

                self.assign_available_permits();
            }
            SearchPermitMessage::FreeWarmupSlot => {
                self.num_warmup_slots_available += 1;
                self.assign_available_permits();
            }
            SearchPermitMessage::Drop {
                memory_size,
                warmup_slot_freed,
            } => {
                // total_job_cost is decremented synchronously in `SearchPermit::Drop`. This eases
                // testing (no need to wait for the queue to drain to observe the cost was updated)
                if !warmup_slot_freed {
                    self.num_warmup_slots_available += 1;
                }
                self.total_memory_allocated = self
                    .total_memory_allocated
                    .checked_sub(memory_size)
                    .expect("More memory released than allocated, should never happen.");
                self.assign_available_permits();
            }
        }
    }

    fn pop_next_request_if_serviceable(&mut self) -> Option<SingleSplitPermitRequest> {
        if self.num_warmup_slots_available == 0 {
            return None;
        }
        let available_memory = self
            .total_memory_budget
            .checked_sub(self.total_memory_allocated)?;
        let mut peeked = self.permits_requests.peek_mut()?;

        assert!(
            !peeked.is_empty(),
            "unexpected empty permits_requests present in the search permit provider queue"
        );

View on GitHub (pinned to a39730c5cd)