quickwit-oss/tantivy · error · io::Error

Fst data is corrupted: {err:?}

Error message

Fst data is corrupted: {err:?}

What it means

Thrown by open_fst_index() (reached via open()) when tantivy_fst::Fst::new rejects the term dictionary's FST bytes. A valid FST has a strict binary format; any parse failure means the term index data is corrupted or was not produced by this library version.

Source

Thrown at src/termdict/fst_termdict/termdict.rs:95

    /// `Write` object.
    pub fn finish(mut self) -> io::Result<W> {
        let mut file = self.fst_builder.into_inner().map_err(convert_fst_error)?;
        {
            let mut counting_writer = CountingWriter::wrap(&mut file);
            self.term_info_store_writer
                .serialize(&mut counting_writer)?;
            let footer_size = counting_writer.written_bytes();
            footer_size.serialize(&mut counting_writer)?;
            FST_VERSION.serialize(&mut counting_writer)?;
        }
        Ok(file)
    }
}

fn open_fst_index(fst_file: FileSlice) -> io::Result<tantivy_fst::Map<OwnedBytes>> {
    let bytes = fst_file.read_bytes()?;
    let fst = Fst::new(bytes).map_err(|err| {
        io::Error::new(
            io::ErrorKind::InvalidData,
            format!("Fst data is corrupted: {err:?}"),
        )
    })?;
    Ok(tantivy_fst::Map::from(fst))
}

static EMPTY_TERM_DICT_FILE: Lazy<FileSlice> = Lazy::new(|| {
    let term_dictionary_data: Vec<u8> = TermDictionaryBuilder::create(Vec::<u8>::new())
        .expect("Creating a TermDictionaryBuilder in a Vec<u8> should never fail")
        .finish()
        .expect("Writing in a Vec<u8> should never fail");
    FileSlice::from(term_dictionary_data)
});

/// The term dictionary contains all of the terms in
/// `tantivy index` in a sorted manner.
///

View on GitHub (pinned to b5d8deb80c)

Solutions

1. Rebuild the index from source documents or restore from a verified backup
2. Confirm all index files were copied completely and match expected sizes/checksums
3. Ensure the tantivy version opening the index can read the version that wrote it
4. Quarantine the corrupted segment file and re-index just that segment if possible

Example fix

// before
let fst = Fst::new(bytes).map_err(|err| io::Error::new(io::ErrorKind::InvalidData, format!("Fst data is corrupted: {err:?}")))?;
// after
let fst = match Fst::new(bytes) {
    Ok(f) => f,
    Err(err) => {
        eprintln!("term dictionary FST corrupted ({err:?}); falling back to re-index");
        return reindex_segment();
    }
};

fn fst_file_looks_valid(fst_file: &FileSlice) -> bool {
    match fst_file.read_bytes() {
        Ok(bytes) => bytes.len() >= 8 && tantivy_fst::raw::Node::read(bytes.as_slice()).is_ok(),
        Err(_) => false,
    }
}

fn is_parseable_fst(bytes: &OwnedBytes) -> bool {
    tantivy_fst::Fst::new(bytes.clone()).is_ok()
}

match TermDictionary::open(fst_file) {
    Err(e) if e.to_string().starts_with("Fst data is corrupted") => {
        eprintln!("term dictionary corrupted; rebuilding segment");
        rebuild_segment();
    }
    Err(e) => return Err(e.into()),
    Ok(dict) => use_dictionary(dict),
}

When it happens

Trigger: Opening a term dictionary whose FST file slice contains invalid bytes: truncated FST, wrong file contents, version mismatch between writer and reader, or disk corruption.

Common situations: Incomplete copy of the index (missing/short .fst-bearing segment files); mixing segments from different tantivy versions; bit rot or failed writes on segment files.

Related errors

• doc store block not completely decompressed, data corruption
• doc store block not completely decompressed, data corruption
• error when reading block in doc store
• err.to_string()
• File corrupted. The file is smaller than Footer::SIZE_IN_BYT

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/9166208758896bd4. Report an issue: GitHub.