quickwit-oss/tantivy · critical

Failed to deserialize terminfoblockmeta

Error message

Failed to deserialize terminfoblockmeta

What it means

`TermInfoStore::get` slices the serialized block metadata at `term_ord / BLOCK_LEN` and deserializes a `TermInfoBlockMeta` with `expect("Failed to deserialize terminfoblockmeta")`. Deserialization failing means the stored metadata bytes cannot be parsed — i.e. the term dictionary's block metadata on disk is corrupt, truncated, or was written by an incompatible format version.

Source

Thrown at src/termdict/fst_termdict/term_info_store.rs:143

        let (len_slice, main_slice) = term_info_store_file.split(16);
        let mut bytes = len_slice.read_bytes()?;
        let len = u64::deserialize(&mut bytes)? as usize;
        let num_terms = u64::deserialize(&mut bytes)? as usize;
        let (block_meta_file, term_info_file) = main_slice.split(len);
        let term_info_bytes = term_info_file.read_bytes()?;
        Ok(TermInfoStore {
            num_terms,
            block_meta_bytes: block_meta_file.read_bytes()?,
            term_info_bytes,
        })
    }

    pub fn get(&self, term_ord: TermOrdinal) -> TermInfo {
        let block_id = (term_ord as usize) / BLOCK_LEN;
        let buffer = self.block_meta_bytes.as_slice();
        let mut block_data: &[u8] = &buffer[block_id * TermInfoBlockMeta::SIZE_IN_BYTES..];
        let term_info_block_data = TermInfoBlockMeta::deserialize(&mut block_data)
            .expect("Failed to deserialize terminfoblockmeta");
        let inner_offset = (term_ord as usize) % BLOCK_LEN;
        if inner_offset == 0 {
            return term_info_block_data.ref_term_info;
        }
        let term_info_data = self.term_info_bytes.as_slice();
        term_info_block_data.deserialize_term_info(
            &term_info_data[term_info_block_data.offset as usize..],
            inner_offset - 1,
        )
    }

    pub fn num_terms(&self) -> usize {
        self.num_terms
    }
}

pub struct TermInfoStoreWriter {
    buffer_block_metas: Vec<u8>,

View on GitHub (pinned to b5d8deb80c)

Solutions

1. Re-index the data from the original source — corrupt term dictionary metadata cannot be repaired in place reliably.
2. Verify the tantivy version used to read matches the one that wrote the index; version-mismatched wire formats will not deserialize.
3. Restore the segment files from backup and confirm all files were copied completely (checksum/size verification).
4. If you control the store construction (e.g. building a hotcache), validate that `block_meta_bytes` boundaries align exactly with `TermInfoBlockMeta::SIZE_IN_BYTES` multiples.

Example fix

// before: reading a segment copied with an incomplete file list
let store = TermInfoStore::open(partially_copied_segment);
let info = store.get(term_ord); // panics on truncated meta

// after: validate file sizes before opening
if block_meta_len % TermInfoBlockMeta::SIZE_IN_BYTES != 0 {
    return Err("corrupt terminfo block meta: re-index");
}
let store = TermInfoStore::open(verified_segment);

fn block_meta_is_aligned(store_bytes: &[u8]) -> bool {
    use tantivy::termdict::TermInfoBlockMeta;
    !store_bytes.is_empty()
        && store_bytes.len() % TermInfoBlockMeta::SIZE_IN_BYTES == 0
}

fn safe_get(store: &TermInfoStore, term_ord: u64, meta_len: usize) -> Option<TermInfo> {
    let block_id = term_ord as usize / tantivy::termdict::BLOCK_LEN;
    if (block_id + 1) * tantivy::termdict::TermInfoBlockMeta::SIZE_IN_BYTES > meta_len {
        return None; // corrupt/truncated meta
    }
    Some(store.get(term_ord))
}

let info = std::panic::catch_unwind(|| store.get(term_ord))
    .map_err(|_| anyhow::anyhow!("corrupt terminfo block meta at ord {} — re-index", term_ord))?;

When it happens

Trigger: Calling `get(term_ord)` with any ordinal once the store's `block_meta_bytes` are corrupt/truncated; opening an index whose termdict was written by a different tantivy version with a different `TermInfoBlockMeta` wire format; or constructing the store with misaligned bytes (wrong offsets in the hotcache/segment layout).

Common situations: Index corruption after a crash or disk failure, copying segments incompletely (missing/truncated files), opening an index across incompatible library versions, or bit-rot on stored segment files.

Related errors

• Corrupted data. Invalid VInt 32
• Invalid op metadata byte
• term dict returned non-UTF-8
• could not convert to String
• TermMissingAgg collector, but no missing found in agg req

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/53213346240f8278. Report an issue: GitHub.