xai-org/x-algorithm · warning · anyhow::Error

MM_LOCAL_SNAPSHOT: failed to read {path}: {e}

Error message

MM_LOCAL_SNAPSHOT: failed to read {path}: {e}

What it means

Raised in get_mm_client when the MM_LOCAL_SNAPSHOT env var is set but the referenced parquet file cannot be read. This is explicitly an offline/dev path (the log warns it skips O2), so the failure means the local snapshot file path is wrong or unreadable.

Source

Thrown at phoenix/crates/serving/xai-recsys-mm-server/src/mm_embedding_client.rs:296

    if !is_writer {
        log::info!(
            "Worker {}: opened MM cache (read-only, {} shards)",
            worker_id,
            NUM_SHARDS
        );
        return Ok((client, Box::pin(async { Ok(()) })));
    }

    if let Ok(path) = env::var("MM_LOCAL_SNAPSHOT")
        && !path.trim().is_empty()
    {
        log::warn!(
            "MM_LOCAL_SNAPSHOT set ({path}): loading MM embeddings from a LOCAL parquet \
             and SKIPPING O2 (offline/dev path, NOT production behavior)."
        );
        let data = std::fs::read(&path)
            .map_err(|e| anyhow::anyhow!("MM_LOCAL_SNAPSHOT: failed to read {path}: {e}"))?;
        let records = read_embeddings_from_parquet(bytes::Bytes::from(data), &path)?;
        let n = records.len();
        let client_for_load = client.clone();
        let load_future = async move {
            client_for_load.preload_embeddings(records, true).await;
            log::warn!("MM_LOCAL_SNAPSHOT: loaded {n} MM embeddings into the cache (NOT O2).");
            Ok(())
        };
        return Ok((client, Box::pin(load_future)));
    }

    let mm_client_for_bg = client.clone();
    let (o2_bucket, o2_prefix) = o2_env_vars();

    match MmEmbeddingFetcher::from_env(&o2_bucket, &o2_prefix) {
        Ok(o2_client) => {
            log::info!("Successfully created O2 client, fetching embeddings for last 2 days...");
            let o2_future = async move {

View on GitHub (pinned to 24c60942c5)

Solutions

1. Verify the path exists and is readable: ls -l $MM_LOCAL_SNAPSHOT
2. Use an absolute path (no ~ or relative paths) in the env var
3. In containers, ensure the snapshot file is mounted/copied into the image or volume
4. Unset MM_LOCAL_SNAPSHOT in environments where the real O2 path should be used

Example fix

# before
export MM_LOCAL_SNAPSHOT=~/snapshots/mm.parquet

# after
export MM_LOCAL_SNAPSHOT=/home/user/snapshots/mm.parquet
ls -l $MM_LOCAL_SNAPSHOT  # verify before launching

if let Ok(p) = std::env::var("MM_LOCAL_SNAPSHOT") { assert!(std::fs::metadata(&p).map(|m| m.is_file()).unwrap_or(false), "snapshot not readable: {p}"); }

null

When it happens

Trigger: Setting MM_LOCAL_SNAPSHOT to a nonexistent path, a path without read permissions, a directory, or a remote/uncached path; the file being locked or deleted mid-run.

Common situations: Dev machines pointing at a stale snapshot path; container deployments where the snapshot wasn't mounted; typos or ~ that isn't expanded; NFS/latency causing read failures.

Related errors

• {ENV_CACHE_WARM_SAMPLE_PCT}: {error}
• {misconfiguration}
• {ENV_GRPC_MTLS_SERVER_KEY_PATH} must be set
• {ENV_GRPC_MTLS_SERVER_CRT_PATH} must be set
• {ENV_GRPC_MTLS_CLIENT_CA_PATH} must be set

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/eae27b7c6faa0773. Report an issue: GitHub.