chroma-core/chroma · error · CacheError

failed to close cache: {:?}

Error message

failed to close cache: {:?}

What it means

FoyerHybridCache::close (the Cache trait impl in rust/cache/src/foyer.rs) awaits foyer::Cache::close and maps any failure into CacheError::DiskError with the anyhow message 'failed to close cache: {:?}'. Closing a foyer cache flushes dirty in-memory entries to disk storage, so this error means the shutdown-time flush/persist step failed — typically an underlying I/O error, a full or read-only disk, or a file handle problem — and it wraps the original foyer error for inspection.

Source

Thrown at rust/cache/src/foyer.rs:529

    V: Clone + Send + Sync + StorageValue + Weighted + 'static,
{
    async fn insert_to_disk(&self, key: K, value: V) {
        let hostname = std::slice::from_ref(&self.hostname);
        let _stopwatch = Stopwatch::new(
            &self.insert_to_disk_latency,
            hostname,
            StopWatchUnit::Millis,
        );
        // Write directly to disk storage, bypassing the memory cache.
        // This is useful for prefetching data that is not immediately needed.
        self.cache.storage_writer(key).insert(value);
    }

    async fn close(&self) -> Result<(), CacheError> {
        self.cache
            .close()
            .await
            .map_err(|e| CacheError::DiskError(anyhow::anyhow!("failed to close cache: {:?}", e)))
    }
}

#[derive(Clone)]
pub struct FoyerPlainCache<K, V>
where
    K: Clone + Send + Sync + Eq + PartialEq + Hash + 'static,
    V: Clone + Send + Sync + Weighted + 'static,
{
    cache: foyer::Cache<K, V>,
    cache_hit: opentelemetry::metrics::Counter<u64>,
    cache_miss: opentelemetry::metrics::Counter<u64>,
    get_latency: opentelemetry::metrics::Histogram<u64>,
    obtain_latency: opentelemetry::metrics::Histogram<u64>,
    insert_latency: opentelemetry::metrics::Histogram<u64>,
    remove_latency: opentelemetry::metrics::Histogram<u64>,
    clear_latency: opentelemetry::metrics::Histogram<u64>,
    hostname: KeyValue,

View on GitHub (pinned to aecdd12c8a)

Solutions

1. Inspect the wrapped error (the {:?} payload) to identify the I/O root cause: log/return the CacheError::DiskError and check errno — ENOSPC, EACCES, EROFS each point to a different fix.
2. Free disk space or fix permissions on the cache directory, then retry close — flushed-then-closed state can usually be reached on the second attempt.
3. Stop all writers (prefetch/storage_writer tasks) and await them before calling close so no inserts race the flush.
4. If data was already persisted incrementally, treat close failure as non-fatal: log, continue shutdown, and accept that the tail of dirty entries is lost; for strict durability, use a cache config with more aggressive write-through/flush cadence.

Example fix

// before
self.cache.close().await?;  // CacheError::DiskError("failed to close cache: ...") propagates and aborts shutdown

# after
if let Err(CacheError::DiskError(e)) = self.cache.close().await {
    tracing::error!(error = ?e, "cache close failed; continuing shutdown");
    // decide policy: retry once, or accept loss of unflushed tail
}

// Before close: make sure the cache directory is writable and the volume has headroom.
use std::path::Path;

fn cache_dir_prepared(dir: &Path) -> bool {
    let Ok(meta) = std::fs::metadata(dir) else { return false };
    meta.is_dir() && std::fs::OpenOptions::new().append(true).create(true)
        .open(dir.join(".close_probe")).map(|f| true).unwrap_or(false)
}

fn is_disk_close_error(err: &CacheError) -> bool {
    matches!(err, CacheError::DiskError(e) if format!("{:?}", e).contains("failed to close cache"))
}

match cache.close().await {
    Ok(()) => tracing::info!("cache closed cleanly"),
    Err(CacheError::DiskError(e)) => {
        // shutdown-time flush failed: log the wrapped I/O cause, optionally retry once,
        // then continue shutdown rather than aborting the process
        tracing::error!(error = ?e, "failed to close cache");
        if retry_close_once(cache).await.is_err() {
            tracing::warn!("unflushed tail entries may be lost");
        }
    }
    Err(other) => return Err(other),
}

When it happens

Trigger: Calling cache.close() during process shutdown or cache teardown and the underlying foyer close returns Err: disk out of space while flushing dirty entries, permission loss on the cache directory, the device/filesystem erroring, or concurrent/misuse of the cache handle during close (e.g. writers still inserting via storage_writer while close runs).

Common situations: Long-running servers shutting down on a full volume after heavy writes; container environments where the cache directory is on an overlay/ephemeral mount that became unwritable; crash-looping pods that close caches repeatedly; tests that drop/close caches while background prefetch writers (storage_writer().insert(...)) are still active.

Related errors

• not implemented
• Embedding function provided when already defined in the coll
• Aggregate input must be an Aggregate instance or object with
• MinK keys cannot be empty
• MinK k must be positive

AI-assisted analysis of chroma-core/chroma@aecdd12c8a (2026-08-16). Data as JSON: /api/errors/8e831fb4d37667e3. Report an issue: GitHub.