{"record":{"id":"e29b135fc869fa2e","repo":"quickwit-oss/tantivy","slug":"invaliddata","errorCode":"InvalidData","errorMessage":"Not valid utf-8","messagePattern":"Not valid utf-8","errorType":"error_code","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"columnar/src/column/dictionary_encoded.rs","lineNumber":113,"sourceCode":"    pub fn wrap(bytes_column: BytesColumn) -> StrColumn {\n        StrColumn(bytes_column)\n    }\n\n    pub fn dictionary(&self) -> &Dictionary<VoidSSTable> {\n        self.0.dictionary.as_ref()\n    }\n\n    /// Fills the buffer\n    pub fn ord_to_str(&self, term_ord: u64, output: &mut String) -> io::Result<bool> {\n        unsafe {\n            let buf = output.as_mut_vec();\n            if !self.0.dictionary.ord_to_term(term_ord, buf)? {\n                return Ok(false);\n            }\n            // TODO consider remove checks if it hurts performance.\n            if std::str::from_utf8(buf.as_slice()).is_err() {\n                buf.clear();\n                return Err(io::Error::new(\n                    io::ErrorKind::InvalidData,\n                    \"Not valid utf-8\",\n                ));\n            }\n        }\n        Ok(true)\n    }\n}\n\nimpl Deref for StrColumn {\n    type Target = BytesColumn;\n\n    fn deref(&self) -> &Self::Target {\n        &self.0\n    }\n}\n","sourceCodeStart":95,"sourceCodeEnd":130,"githubUrl":"https://github.com/quickwit-oss/tantivy/blob/b5d8deb80c26924e6b007a5b1a7630f35ca64de4/columnar/src/column/dictionary_encoded.rs#L95-L130","documentation":"ord_to_str resolves a term ordinal in a dictionary-encoded column back to its string term, which must be valid UTF-8. If the underlying term bytes are not valid UTF-8, the method clears the buffer and returns an io::Error with kind InvalidData. This indicates corrupted or non-string data in a column expected to be string-typed.","triggerScenarios":"Calling ord_to_str on a DictionaryEncodedColumn whose dictionary contains non-UTF-8 terms — e.g. after reading a corrupted index, wrong column type interpretation, or terms written by an older/incompatible writer.","commonSituations":"Reading index files from a different tantivy/columnar version; corrupted hotcache/segment files; manually serialized columns mixing binary terms into string dictionaries.","solutions":["Validate the column is a string column before calling ord_to_str; use the term-oriented API for binary columns.","Re-index the segment / rebuild the index if files are corrupted.","Ensure reader and writer versions (format_version) match; migrate data instead of cross-version reads.","Check filesystem integrity / disk errors causing corruption."],"exampleFix":"// before\nlet s = column.ord_to_str(ord)?; // InvalidData if term is not utf-8\n// after\nlet mut buf = String::new();\nif column.ord_to_str(ord, &mut buf)? {\n    let s: &str = &buf; // only used when valid\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":"fn is_valid_utf8_term(buf: &[u8]) -> bool {\n    std::str::from_utf8(buf).is_ok()\n}","tryCatchPattern":"match column.ord_to_str(ord, &mut buf) {\n    Err(e) if e.kind() == std::io::ErrorKind::InvalidData => {\n        // treat as corrupt column: skip term or rebuild index\n    }\n    other => other?,\n}","preventionTips":["Avoid mixing binary terms into string dictionary columns.","Pin compatible tantivy versions for writer and reader.","Verify segment file checksums after copying/downloading indexes.","Fall back to term APIs rather than forcing ord_to_str on unknown columns."],"tags":["io","utf8","dictionary","corruption"],"backgroundTag":"invalid-utf8-term","analyzedSha":"b5d8deb80c26924e6b007a5b1a7630f35ca64de4","analyzedAt":"2026-09-05T13:20:51.521Z","contentChangedAt":"2026-09-05T13:20:51.521Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}