quickwit-oss/tantivy · error · io::Error

SSTable corruption

Error message

SSTable corruption

What it means

When opening an SSTable with format version 2, the serialized index footer/index bytes fail to parse (v2::SSTableIndex::load returns Err), so open() converts that into io::Error of kind InvalidData with the message "SSTable corruption". It means the index portion of the file is not valid for the declared version — usually the file is truncated, partially written, or not actually the version it claims.

Source

Thrown at sstable/src/index/mod.rs:28

use crate::{TermOrdinal, common_prefix_len};

#[derive(Debug, Clone)]
pub enum SSTableIndex {
    V2(v2::SSTableIndex),
    V3(v3::SSTableIndexV3),
    V3Empty(v3::SSTableIndexV3Empty),
}

impl SSTableIndex {
    pub(crate) fn open(
        version: u32,
        index_offset: u64,
        index_bytes: OwnedBytes,
    ) -> io::Result<Self> {
        let index = match version {
            2 => {
                SSTableIndex::V2(v2::SSTableIndex::load(index_bytes).map_err(|_| {
                    io::Error::new(io::ErrorKind::InvalidData, "SSTable corruption")
                })?)
            }
            3 => {
                let (index_bytes, mut footerv3_len_bytes) = index_bytes.rsplit(8);
                let store_offset = u64::deserialize(&mut footerv3_len_bytes)?;
                if store_offset != 0 {
                    SSTableIndex::V3(v3::SSTableIndexV3::load(index_bytes, store_offset).map_err(
                        |_| io::Error::new(io::ErrorKind::InvalidData, "SSTable corruption"),
                    )?)
                } else {
                    // if store_offset is zero, there is no index, so we build a pseudo-index
                    // assuming a single block of sstable covering everything.
                    SSTableIndex::V3Empty(v3::SSTableIndexV3Empty::load(index_offset as usize))
                }
            }
            _ => {
                return Err(io::Error::other(format!(
                    "Unsupported sstable version, expected one of [2, 3], found {version}"

View on GitHub (pinned to b5d8deb80c)

Solutions

1. Verify the segment file is complete and untruncated; restore from backup or re-index the affected segment.
2. Re-check how index_offset/index_bytes are derived — a wrong offset yields misaligned bytes that fail to load.
3. Ensure the reading library version matches the version that wrote the index; upgrade tantivy if the index was written by a newer release.
4. Run a corruption check / segment merge to rebuild the index from the stored docs if supported.

// Before opening: check file size covers the expected index region
fn index_bytes_intact(file_len: u64, index_offset: u64, min_index_len: u64) -> bool {
    file_len >= index_offset.saturating_add(min_index_len)
}

// catch InvalidData "SSTable corruption" and fall back to re-indexing
match SSTable::open(...) {
    Err(e) if e.kind() == io::ErrorKind::InvalidData
        && e.to_string() == "SSTable corruption" => {
        // restore from backup or rebuild segment
    }
    other => other?,
}

When it happens

Trigger: Calling SSTable::open on a version-2 sstable whose index_bytes cannot be deserialized by v2::SSTableIndex::load: truncated file, wrong offset passed for index_bytes, or bytes from a different/newer format labeled as v2.

Common situations: Interrupted index writes (crash/power loss) leaving a partial segment; copying index files without the footer bytes; opening files produced by an incompatible tantivy version; corrupted downloads or bad mmap ranges.

Related errors

• failed to read block_len
• failed to read block content
• Corrupted data. Invalid VInt 32
• InvalidData
• UnexpectedEof

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/1314cdf357e47305. Report an issue: GitHub.