{"record":{"id":"b8cafdc3009783bb","repo":"influxdata/influxdb","slug":"never-fails","errorCode":null,"errorMessage":"never fails","messagePattern":"never fails","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"core/object_store_mem_cache/src/cache_system/s3_fifo_cache/mod.rs","lineNumber":231,"sourceCode":"            let fetch_res = match fetch_res {\n                Ok(v) => {\n                    if let Some(cache) = cache_captured.upgrade() {\n                        // NOTES:\n                        // - Don't involve hook here because the cache is doing that for us correctly, even if the key is\n                        //   already stored.\n                        // - Tell tokio that this is potentially expensive. This is due to the fact that inserting new values\n                        //   may free existing ones and the relevant allocator accounting can be rather pricey.\n                        // - We pass `v` by value because there is the small chance that between checking the S3-FIFO\n                        //   and creating this loader future, the S3-FIFO might have been updated. In that case\n                        //   `S3Fifo::get_or_put` will return the exiting entry, but also the to-be-inserted (that we\n                        //   originally fetched here) one as \"to be evicted\" (so we don't have two copies).\n                        // - If the cache is full and all entries are in-use, get_or_put returns Err with the value.\n                        //   In that case, we return the fetched value without caching it.\n                        let k = Arc::clone(&k_captured);\n                        let result =\n                            tokio::task::spawn_blocking(move || cache.get_or_put(k, v, generation))\n                                .await\n                                .expect(\"never fails\");\n\n                        match result {\n                            Ok((entry, evicted)) => {\n                                evicted.async_drop().await;\n                                Ok(entry.value().clone())\n                            }\n                            Err((value, evicted)) => {\n                                // Balance hook.fetched that was called before eviction failed\n                                hook_captured.evict(\n                                    generation,\n                                    &k_captured,\n                                    EvictResult::Fetched { size: value.size() },\n                                );\n                                evicted.async_drop().await;\n\n                                value.async_drop().await;\n                                let msg: DynError = Arc::new(EntriesInUseError);\n","sourceCodeStart":213,"sourceCodeEnd":249,"githubUrl":"https://github.com/influxdata/influxdb/blob/d28e26e048401c53cbb98cf2d6ab0cf1e98048ca/core/object_store_mem_cache/src/cache_system/s3_fifo_cache/mod.rs#L213-L249","documentation":"The S3-FIFO in-memory cache inserts fetched values via tokio::task::spawn_blocking(move || cache.get_or_put(k, v, generation)) and .await.expect(\"never fails\") on the JoinHandle. That expect only fires if the blocking task itself fails: a panic inside get_or_put (cache accounting/invariant bug, internal lock poisoned) or the task being aborted at runtime shutdown. A full cache is NOT this error - get_or_put returns Ok/Err normally for that, and only the Err arm of `result` handles eviction failure.","triggerScenarios":"A panic inside S3Fifo::get_or_put on the blocking thread (capacity or accounting invariant broken, poisoned internal lock); dropping/shutting down the tokio runtime while a spawned cache insert is still in flight, so awaiting the JoinHandle yields a JoinError.","commonSituations":"Memory-cache size configured too small relative to entry sizes, stressing eviction/accounting paths; cache implementation bugs; integration tests that tear down the runtime with pending cache activity.","solutions":["Look for the preceding panic message/backtrace - the JoinError embeds the original panic text; that is the real bug","Review mem-cache-size and cache dimension settings against typical object sizes","Shut down the object store/cache gracefully before dropping the runtime so no spawn_blocking insert is orphaned","If it reproduces on a released influxdb3 build without local patches, capture the panic and file an issue against object_store_mem_cache"],"exampleFix":"// before\nlet result = tokio::task::spawn_blocking(move || cache.get_or_put(k, v, generation))\n    .await\n    .expect(\"never fails\");\n\n// after: surface the join failure instead of panicking\nlet result = tokio::task::spawn_blocking(move || cache.get_or_put(k, v, generation))\n    .await\n    .map_err(|e| anyhow::anyhow!(\"S3-FIFO insert task failed: {e}\"))?;\n","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// if you maintain this code: handle the join failure explicitly\nlet result = tokio::task::spawn_blocking(move || cache.get_or_put(k, v, generation))\n    .await\n    .map_err(|e| anyhow::anyhow!(\"S3-FIFO insert task failed (panic or shutdown): {e}\"))?;\n","preventionTips":["Size mem-cache appropriately for your object sizes so eviction paths are not pathological","Shut down the store/cache before dropping the tokio runtime in tests and binaries","When it fires, read the embedded panic text - the real bug is inside get_or_put, not this expect"],"tags":["cache","s3-fifo","spawn-blocking","join-error","panic","influxdb3"],"backgroundTag":"blocking-task-panic","analyzedSha":"d28e26e048401c53cbb98cf2d6ab0cf1e98048ca","analyzedAt":"2026-08-16T19:53:34.623Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}