influxdata/influxdb · error

ghost queue is NOT empty

Error message

ghost queue is NOT empty

What it means

LockedState::insert_ghost trims the ghost queue with `while self.ghost.memory_size() >= config.max_ghost_memory_size { pop_front().expect("ghost queue is NOT empty") }`. The invariant is that a non-empty budget implies a non-empty queue, but with max_ghost_memory_size == 0 an EMPTY queue has memory_size() == 0 >= 0, so pop_front() is called on an empty OrderedSet, returns None, and the expect panics. Unlike the public store builder (which uses NonZeroUsize), the S3Config field is a plain pub usize, so a zero ghost budget is representable and immediately fatal.

Source

Thrown at core/object_store_mem_cache/src/cache_system/s3_fifo_cache/s3_fifo.rs:810

/// their keys, and an [EvictionResult] indicating success or failure.
struct EvictionAttempt<K, V>
where
    K: ?Sized,
{
    entries: Vec<CacheEntry<K, V>>,
    keys: Vec<Arc<K>>,
}

type EvictionResult<K, V> = Result<EvictionAttempt<K, V>, EvictionAttempt<K, V>>;

impl<K, V> LockedState<K, V>
where
    K: Debug + Eq + Hash + HasSize + Send + Sync + 'static + ?Sized,
    V: HasSize + InUse + Send + Sync + 'static,
{
    fn insert_ghost(&mut self, key: Arc<K>, config: &S3Config<K>, evicted_keys: &mut Vec<Arc<K>>) {
        while self.ghost.memory_size() >= config.max_ghost_memory_size {
            evicted_keys.push(self.ghost.pop_front().expect("ghost queue is NOT empty"));
        }

        self.ghost.push_back(key);
    }

    /// Evict entries, returning an [EvictionAttempt] with the result.
    fn evict(&mut self, entries: &Entries<K, V>, config: &S3Config<K>) -> EvictionResult<K, V> {
        let small_queue_threshold =
            (config.move_to_main_threshold * config.max_memory_size as f64) as usize;
        let mut evicted_entries = Vec::with_capacity(8);
        let mut evicted_keys = Vec::with_capacity(8);

        while self.small.memory_size() + self.main.memory_size() >= config.max_memory_size {
            if self.small.memory_size() >= small_queue_threshold {
                match self.evict_from_small_queue(
                    entries,
                    config,
                    &mut evicted_entries,

View on GitHub (pinned to d28e26e048)

Solutions

1. Set max_ghost_memory_size to a positive value (e.g. usize::MAX or a real byte budget) — 0 is invalid, not 'disabled'.
2. If you build the cache through MemCacheObjectStoreParams, keep s3_fifo_ghost_memory_limit as NonZeroUsize so 0 is unrepresentable.
3. Add a debug_assert!/validation in S3Config construction that max_ghost_memory_size > 0 (upstream fix worth filing).
4. Audit generated S3Config literals (tests, benches) for zero ghost budgets before reusing them.

Example fix

// before
let config = S3Config {
    max_ghost_memory_size: 0, // "disable" ghost queue -> panics in insert_ghost
    ..
};

// after
let config = S3Config {
    max_ghost_memory_size: usize::MAX, // effectively unbounded ghost set
    ..
};

// validate S3Config before constructing the cache
fn valid_ghost_budget(cfg: &S3Config<impl HasSize>) -> bool {
    cfg.max_ghost_memory_size > 0 // 0 makes insert_ghost pop from an empty queue
}

if !valid_ghost_budget(&config) {
    return Err("max_ghost_memory_size must be > 0 (use usize::MAX for unbounded)".into());
}
let cache = S3FifoCache::new(config, &registry);

// make an invalid budget unrepresentable at your API boundary
fn ghost_budget(n: usize) -> Option<NonZeroUsize> {
    NonZeroUsize::new(n)
}

When it happens

Trigger: Constructing S3Config { max_ghost_memory_size: 0, .. } (or any ghost budget of 0 via direct struct literal / tests) and performing any insert that promotes an evicted key into the ghost set — insert_ghost is called on the first eviction from the small queue.

Common situations: Disabling the ghost queue by setting its byte limit to 0 in config (a natural but wrong way to 'turn it off' — use usize::MAX instead); copying test S3Config literals into production code paths; config plumbing that maps an unset/0 default to the raw field, bypassing the NonZeroUsize guard that MemCacheObjectStoreParams enforces.

Related errors

• not poisoned
• not poisoned
• num_row_groups_in_parallel should be above zero
• should exist
• just got this index

AI-assisted analysis of influxdata/influxdb@d28e26e048 (2026-08-16). Data as JSON: /api/errors/93cdc76ab3d53ca4. Report an issue: GitHub.