xai-org/x-algorithm · error · anyhow::Error

EmbeddingCache: missing emb_dim metadata

Error message

EmbeddingCache: missing emb_dim metadata

What it means

Raised by EmbeddingCache::open when the LMDB database exists but the meta key LMDB_META_KEY_EMB_DIM is absent. The embedding dimension is required to compute slot size (SLOT_HEADER_SIZE + emb_dim * sizeof(f16)) for the mmap region, so the cache cannot be laid out.

Source

Thrown at phoenix/crates/serving/xai-recsys-mm-server/src/embedding_cache.rs:284

        let data_mdb = lmdb_path.join("data.mdb");
        let map_size = std::fs::metadata(&data_mdb)
            .map(|m| m.len() as usize)
            .unwrap_or(0)
            .max(1024 * 1024);

        let env = unsafe {
            heed::EnvOpenOptions::new()
                .map_size(map_size)
                .max_readers(1024)
                .open(lmdb_path)?
        };
        let rtxn = env.read_txn()?;
        let db = env.open_database(&rtxn, None)?.ok_or_else(|| {
            anyhow::anyhow!("EmbeddingCache: no database in {}", lmdb_path.display())
        })?;
        let emb_dim = db
            .get(&rtxn, &LMDB_META_KEY_EMB_DIM)?
            .ok_or_else(|| anyhow::anyhow!("EmbeddingCache: missing emb_dim metadata"))?
            as usize;
        rtxn.commit()?;

        let file = OpenOptions::new().read(true).write(true).open(data_path)?;
        let mmap = unsafe { MmapMut::map_mut(&file)? };
        let slot_size = SLOT_HEADER_SIZE + emb_dim * std::mem::size_of::<f16>();
        let capacity = (mmap.len() - HEADER_SIZE) / slot_size;

        log::info!(
            "EmbeddingCache: opened lmdb={} data={} (capacity={}, emb_dim={})",
            lmdb_path.display(),
            data_path.display(),
            capacity,
            emb_dim,
        );

        Ok(Self {
            env,

View on GitHub (pinned to 24c60942c5)

Solutions

1. Regenerate the cache with the current version of create() so emb_dim metadata is written
2. If you control the writer, ensure it stores LMDB_META_KEY_EMB_DIM before committing
3. Pass emb_dim explicitly or add a fallback (infer from data file size / slot size) if regeneration is too costly
4. Add a cache-format version key alongside emb_dim to detect format mismatches early

Example fix

// before
let emb_dim = db.get(&rtxn, &LMDB_META_KEY_EMB_DIM)?
    .ok_or_else(|| anyhow::anyhow!("EmbeddingCache: missing emb_dim metadata"))? as usize;

// after
let emb_dim = match db.get(&rtxn, &LMDB_META_KEY_EMB_DIM)? {
    Some(v) => v as usize,
    None => {
        let slot = data_file_size / num_slots; // fallback inference
        if slot > SLOT_HEADER_SIZE { (slot - SLOT_HEADER_SIZE) / std::mem::size_of::<f16>() }
        else { anyhow::bail!("EmbeddingCache: missing emb_dim metadata and cannot infer") }
    }
};

// before open(): check meta key presence with a quick read_txn; if absent, regenerate cache

null

When it happens

Trigger: Calling open() on an LMDB database that was populated by an older version of create() that didn't write emb_dim metadata, or by an external writer; partially-written/corrupt metadata.

Common situations: Version skew between the tool that wrote the cache and the one reading it; interrupted create() that wrote records but not metadata; manual LMDB population without the meta key.

Related errors

• EmbeddingCache: no database in {}
• Unknown tag %d
• Failed to create serializer for ThriftStruct %s: %s
• Cannot load metadata from {metadata_path}
• No partitions found for topic '{topic}'. Available topics: {

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/231b70641ebc12f1. Report an issue: GitHub.