influxdata/influxdb · critical

never fails

Error message

never fails

What it means

The S3-FIFO in-memory cache inserts fetched values via tokio::task::spawn_blocking(move || cache.get_or_put(k, v, generation)) and .await.expect("never fails") on the JoinHandle. That expect only fires if the blocking task itself fails: a panic inside get_or_put (cache accounting/invariant bug, internal lock poisoned) or the task being aborted at runtime shutdown. A full cache is NOT this error - get_or_put returns Ok/Err normally for that, and only the Err arm of `result` handles eviction failure.

Source

Thrown at core/object_store_mem_cache/src/cache_system/s3_fifo_cache/mod.rs:231

            let fetch_res = match fetch_res {
                Ok(v) => {
                    if let Some(cache) = cache_captured.upgrade() {
                        // NOTES:
                        // - Don't involve hook here because the cache is doing that for us correctly, even if the key is
                        //   already stored.
                        // - Tell tokio that this is potentially expensive. This is due to the fact that inserting new values
                        //   may free existing ones and the relevant allocator accounting can be rather pricey.
                        // - We pass `v` by value because there is the small chance that between checking the S3-FIFO
                        //   and creating this loader future, the S3-FIFO might have been updated. In that case
                        //   `S3Fifo::get_or_put` will return the exiting entry, but also the to-be-inserted (that we
                        //   originally fetched here) one as "to be evicted" (so we don't have two copies).
                        // - If the cache is full and all entries are in-use, get_or_put returns Err with the value.
                        //   In that case, we return the fetched value without caching it.
                        let k = Arc::clone(&k_captured);
                        let result =
                            tokio::task::spawn_blocking(move || cache.get_or_put(k, v, generation))
                                .await
                                .expect("never fails");

                        match result {
                            Ok((entry, evicted)) => {
                                evicted.async_drop().await;
                                Ok(entry.value().clone())
                            }
                            Err((value, evicted)) => {
                                // Balance hook.fetched that was called before eviction failed
                                hook_captured.evict(
                                    generation,
                                    &k_captured,
                                    EvictResult::Fetched { size: value.size() },
                                );
                                evicted.async_drop().await;

                                value.async_drop().await;
                                let msg: DynError = Arc::new(EntriesInUseError);

View on GitHub (pinned to d28e26e048)

Solutions

1. Look for the preceding panic message/backtrace - the JoinError embeds the original panic text; that is the real bug
2. Review mem-cache-size and cache dimension settings against typical object sizes
3. Shut down the object store/cache gracefully before dropping the runtime so no spawn_blocking insert is orphaned
4. If it reproduces on a released influxdb3 build without local patches, capture the panic and file an issue against object_store_mem_cache

Example fix

// before
let result = tokio::task::spawn_blocking(move || cache.get_or_put(k, v, generation))
    .await
    .expect("never fails");

// after: surface the join failure instead of panicking
let result = tokio::task::spawn_blocking(move || cache.get_or_put(k, v, generation))
    .await
    .map_err(|e| anyhow::anyhow!("S3-FIFO insert task failed: {e}"))?;

// if you maintain this code: handle the join failure explicitly
let result = tokio::task::spawn_blocking(move || cache.get_or_put(k, v, generation))
    .await
    .map_err(|e| anyhow::anyhow!("S3-FIFO insert task failed (panic or shutdown): {e}"))?;

When it happens

Trigger: A panic inside S3Fifo::get_or_put on the blocking thread (capacity or accounting invariant broken, poisoned internal lock); dropping/shutting down the tokio runtime while a spawned cache insert is still in flight, so awaiting the JoinHandle yields a JoinError.

Common situations: Memory-cache size configured too small relative to entry sizes, stressing eviction/accounting paths; cache implementation bugs; integration tests that tear down the runtime with pending cache activity.

Related errors

• can only accept once
• not poisoned
• cannot fit duration into u64
• metric should be in progress
• no metric in progress

AI-assisted analysis of influxdata/influxdb@d28e26e048 (2026-08-16). Data as JSON: /api/errors/b8cafdc3009783bb. Report an issue: GitHub.