{"record":{"id":"9166208758896bd4","repo":"quickwit-oss/tantivy","slug":"fst-data-is-corrupted-err","errorCode":null,"errorMessage":"Fst data is corrupted: {err:?}","messagePattern":"Fst data is corrupted: (.+?)","errorType":"exception","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"src/termdict/fst_termdict/termdict.rs","lineNumber":95,"sourceCode":"    /// `Write` object.\n    pub fn finish(mut self) -> io::Result<W> {\n        let mut file = self.fst_builder.into_inner().map_err(convert_fst_error)?;\n        {\n            let mut counting_writer = CountingWriter::wrap(&mut file);\n            self.term_info_store_writer\n                .serialize(&mut counting_writer)?;\n            let footer_size = counting_writer.written_bytes();\n            footer_size.serialize(&mut counting_writer)?;\n            FST_VERSION.serialize(&mut counting_writer)?;\n        }\n        Ok(file)\n    }\n}\n\nfn open_fst_index(fst_file: FileSlice) -> io::Result<tantivy_fst::Map<OwnedBytes>> {\n    let bytes = fst_file.read_bytes()?;\n    let fst = Fst::new(bytes).map_err(|err| {\n        io::Error::new(\n            io::ErrorKind::InvalidData,\n            format!(\"Fst data is corrupted: {err:?}\"),\n        )\n    })?;\n    Ok(tantivy_fst::Map::from(fst))\n}\n\nstatic EMPTY_TERM_DICT_FILE: Lazy<FileSlice> = Lazy::new(|| {\n    let term_dictionary_data: Vec<u8> = TermDictionaryBuilder::create(Vec::<u8>::new())\n        .expect(\"Creating a TermDictionaryBuilder in a Vec<u8> should never fail\")\n        .finish()\n        .expect(\"Writing in a Vec<u8> should never fail\");\n    FileSlice::from(term_dictionary_data)\n});\n\n/// The term dictionary contains all of the terms in\n/// `tantivy index` in a sorted manner.\n///","sourceCodeStart":77,"sourceCodeEnd":113,"githubUrl":"https://github.com/quickwit-oss/tantivy/blob/b5d8deb80c26924e6b007a5b1a7630f35ca64de4/src/termdict/fst_termdict/termdict.rs#L77-L113","documentation":"Thrown by open_fst_index() (reached via open()) when tantivy_fst::Fst::new rejects the term dictionary's FST bytes. A valid FST has a strict binary format; any parse failure means the term index data is corrupted or was not produced by this library version.","triggerScenarios":"Opening a term dictionary whose FST file slice contains invalid bytes: truncated FST, wrong file contents, version mismatch between writer and reader, or disk corruption.","commonSituations":"Incomplete copy of the index (missing/short .fst-bearing segment files); mixing segments from different tantivy versions; bit rot or failed writes on segment files.","solutions":["Rebuild the index from source documents or restore from a verified backup","Confirm all index files were copied completely and match expected sizes/checksums","Ensure the tantivy version opening the index can read the version that wrote it","Quarantine the corrupted segment file and re-index just that segment if possible"],"exampleFix":"// before\nlet fst = Fst::new(bytes).map_err(|err| io::Error::new(io::ErrorKind::InvalidData, format!(\"Fst data is corrupted: {err:?}\")))?;\n// after\nlet fst = match Fst::new(bytes) {\n    Ok(f) => f,\n    Err(err) => {\n        eprintln!(\"term dictionary FST corrupted ({err:?}); falling back to re-index\");\n        return reindex_segment();\n    }\n};","handlingStrategy":"try-catch","validationCode":"fn fst_file_looks_valid(fst_file: &FileSlice) -> bool {\n    match fst_file.read_bytes() {\n        Ok(bytes) => bytes.len() >= 8 && tantivy_fst::raw::Node::read(bytes.as_slice()).is_ok(),\n        Err(_) => false,\n    }\n}","typeGuard":"fn is_parseable_fst(bytes: &OwnedBytes) -> bool {\n    tantivy_fst::Fst::new(bytes.clone()).is_ok()\n}","tryCatchPattern":"match TermDictionary::open(fst_file) {\n    Err(e) if e.to_string().starts_with(\"Fst data is corrupted\") => {\n        eprintln!(\"term dictionary corrupted; rebuilding segment\");\n        rebuild_segment();\n    }\n    Err(e) => return Err(e.into()),\n    Ok(dict) => use_dictionary(dict),\n}","preventionTips":["Verify all segment files transferred completely (checksums, sizes)","Keep writer and reader tantivy versions compatible","Treat index directories as immutable; never patch files in place","Schedule periodic integrity checks on long-lived indexes"],"tags":["io","data-corruption","fst","term-dictionary","rust"],"backgroundTag":"fst-corrupted","analyzedSha":"b5d8deb80c26924e6b007a5b1a7630f35ca64de4","analyzedAt":"2026-09-05T13:20:51.521Z","contentChangedAt":"2026-09-05T13:20:51.521Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}