{"record":{"id":"8e831fb4d37667e3","repo":"chroma-core/chroma","slug":"failed-to-close-cache","errorCode":null,"errorMessage":"failed to close cache: {:?}","messagePattern":"failed to close cache: (.+?)","errorType":"exception","errorClass":"CacheError","httpStatus":null,"severity":"error","filePath":"rust/cache/src/foyer.rs","lineNumber":529,"sourceCode":"    V: Clone + Send + Sync + StorageValue + Weighted + 'static,\n{\n    async fn insert_to_disk(&self, key: K, value: V) {\n        let hostname = std::slice::from_ref(&self.hostname);\n        let _stopwatch = Stopwatch::new(\n            &self.insert_to_disk_latency,\n            hostname,\n            StopWatchUnit::Millis,\n        );\n        // Write directly to disk storage, bypassing the memory cache.\n        // This is useful for prefetching data that is not immediately needed.\n        self.cache.storage_writer(key).insert(value);\n    }\n\n    async fn close(&self) -> Result<(), CacheError> {\n        self.cache\n            .close()\n            .await\n            .map_err(|e| CacheError::DiskError(anyhow::anyhow!(\"failed to close cache: {:?}\", e)))\n    }\n}\n\n#[derive(Clone)]\npub struct FoyerPlainCache<K, V>\nwhere\n    K: Clone + Send + Sync + Eq + PartialEq + Hash + 'static,\n    V: Clone + Send + Sync + Weighted + 'static,\n{\n    cache: foyer::Cache<K, V>,\n    cache_hit: opentelemetry::metrics::Counter<u64>,\n    cache_miss: opentelemetry::metrics::Counter<u64>,\n    get_latency: opentelemetry::metrics::Histogram<u64>,\n    obtain_latency: opentelemetry::metrics::Histogram<u64>,\n    insert_latency: opentelemetry::metrics::Histogram<u64>,\n    remove_latency: opentelemetry::metrics::Histogram<u64>,\n    clear_latency: opentelemetry::metrics::Histogram<u64>,\n    hostname: KeyValue,","sourceCodeStart":511,"sourceCodeEnd":547,"githubUrl":"https://github.com/chroma-core/chroma/blob/aecdd12c8a891610db8653630b066b32ceb678b5/rust/cache/src/foyer.rs#L511-L547","documentation":"FoyerHybridCache::close (the Cache trait impl in rust/cache/src/foyer.rs) awaits foyer::Cache::close and maps any failure into CacheError::DiskError with the anyhow message 'failed to close cache: {:?}'. Closing a foyer cache flushes dirty in-memory entries to disk storage, so this error means the shutdown-time flush/persist step failed — typically an underlying I/O error, a full or read-only disk, or a file handle problem — and it wraps the original foyer error for inspection.","triggerScenarios":"Calling cache.close() during process shutdown or cache teardown and the underlying foyer close returns Err: disk out of space while flushing dirty entries, permission loss on the cache directory, the device/filesystem erroring, or concurrent/misuse of the cache handle during close (e.g. writers still inserting via storage_writer while close runs).","commonSituations":"Long-running servers shutting down on a full volume after heavy writes; container environments where the cache directory is on an overlay/ephemeral mount that became unwritable; crash-looping pods that close caches repeatedly; tests that drop/close caches while background prefetch writers (storage_writer().insert(...)) are still active.","solutions":["Inspect the wrapped error (the {:?} payload) to identify the I/O root cause: log/return the CacheError::DiskError and check errno — ENOSPC, EACCES, EROFS each point to a different fix.","Free disk space or fix permissions on the cache directory, then retry close — flushed-then-closed state can usually be reached on the second attempt.","Stop all writers (prefetch/storage_writer tasks) and await them before calling close so no inserts race the flush.","If data was already persisted incrementally, treat close failure as non-fatal: log, continue shutdown, and accept that the tail of dirty entries is lost; for strict durability, use a cache config with more aggressive write-through/flush cadence."],"exampleFix":"// before\nself.cache.close().await?;  // CacheError::DiskError(\"failed to close cache: ...\") propagates and aborts shutdown\n\n# after\nif let Err(CacheError::DiskError(e)) = self.cache.close().await {\n    tracing::error!(error = ?e, \"cache close failed; continuing shutdown\");\n    // decide policy: retry once, or accept loss of unflushed tail\n}","handlingStrategy":"try-catch","validationCode":"// Before close: make sure the cache directory is writable and the volume has headroom.\nuse std::path::Path;\n\nfn cache_dir_prepared(dir: &Path) -> bool {\n    let Ok(meta) = std::fs::metadata(dir) else { return false };\n    meta.is_dir() && std::fs::OpenOptions::new().append(true).create(true)\n        .open(dir.join(\".close_probe\")).map(|f| true).unwrap_or(false)\n}","typeGuard":"fn is_disk_close_error(err: &CacheError) -> bool {\n    matches!(err, CacheError::DiskError(e) if format!(\"{:?}\", e).contains(\"failed to close cache\"))\n}","tryCatchPattern":"match cache.close().await {\n    Ok(()) => tracing::info!(\"cache closed cleanly\"),\n    Err(CacheError::DiskError(e)) => {\n        // shutdown-time flush failed: log the wrapped I/O cause, optionally retry once,\n        // then continue shutdown rather than aborting the process\n        tracing::error!(error = ?e, \"failed to close cache\");\n        if retry_close_once(cache).await.is_err() {\n            tracing::warn!(\"unflushed tail entries may be lost\");\n        }\n    }\n    Err(other) => return Err(other),\n}","preventionTips":["Quiesce all writers (storage_writer/prefetch tasks) and await them before calling close().","Monitor disk space on the cache volume and alert well before ENOSPC; close-time flush is the first casualty of a full disk.","Treat close() as fallible in shutdown handlers: log-and-continue beats aborting a server mid-shutdown.","Configure foyer with incremental flush/write-through if losing the dirty tail on failed close is unacceptable."],"tags":["rust","foyer","cache","shutdown","disk","io","chroma"],"backgroundTag":"disk-cache-close-failure","analyzedSha":"aecdd12c8a891610db8653630b066b32ceb678b5","analyzedAt":"2026-08-16T21:53:27.228Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}