quickwit-oss/tantivy · error · io::Error

InvalidData

InvalidData

Error message

Not valid utf-8

What it means

ord_to_str resolves a term ordinal in a dictionary-encoded column back to its string term, which must be valid UTF-8. If the underlying term bytes are not valid UTF-8, the method clears the buffer and returns an io::Error with kind InvalidData. This indicates corrupted or non-string data in a column expected to be string-typed.

Source

Thrown at columnar/src/column/dictionary_encoded.rs:113

    pub fn wrap(bytes_column: BytesColumn) -> StrColumn {
        StrColumn(bytes_column)
    }

    pub fn dictionary(&self) -> &Dictionary<VoidSSTable> {
        self.0.dictionary.as_ref()
    }

    /// Fills the buffer
    pub fn ord_to_str(&self, term_ord: u64, output: &mut String) -> io::Result<bool> {
        unsafe {
            let buf = output.as_mut_vec();
            if !self.0.dictionary.ord_to_term(term_ord, buf)? {
                return Ok(false);
            }
            // TODO consider remove checks if it hurts performance.
            if std::str::from_utf8(buf.as_slice()).is_err() {
                buf.clear();
                return Err(io::Error::new(
                    io::ErrorKind::InvalidData,
                    "Not valid utf-8",
                ));
            }
        }
        Ok(true)
    }
}

impl Deref for StrColumn {
    type Target = BytesColumn;

    fn deref(&self) -> &Self::Target {
        &self.0
    }
}

View on GitHub (pinned to b5d8deb80c)

Solutions

1. Validate the column is a string column before calling ord_to_str; use the term-oriented API for binary columns.
2. Re-index the segment / rebuild the index if files are corrupted.
3. Ensure reader and writer versions (format_version) match; migrate data instead of cross-version reads.
4. Check filesystem integrity / disk errors causing corruption.

Example fix

// before
let s = column.ord_to_str(ord)?; // InvalidData if term is not utf-8
// after
let mut buf = String::new();
if column.ord_to_str(ord, &mut buf)? {
    let s: &str = &buf; // only used when valid
}

fn is_valid_utf8_term(buf: &[u8]) -> bool {
    std::str::from_utf8(buf).is_ok()
}

match column.ord_to_str(ord, &mut buf) {
    Err(e) if e.kind() == std::io::ErrorKind::InvalidData => {
        // treat as corrupt column: skip term or rebuild index
    }
    other => other?,
}

When it happens

Trigger: Calling ord_to_str on a DictionaryEncodedColumn whose dictionary contains non-UTF-8 terms — e.g. after reading a corrupted index, wrong column type interpretation, or terms written by an older/incompatible writer.

Common situations: Reading index files from a different tantivy/columnar version; corrupted hotcache/segment files; manually serialized columns mixing binary terms into string dictionaries.

Related errors

• Corrupted data. Invalid VInt 32
• UnexpectedEof
• InvalidData
• InvalidData
• InvalidData

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/e29b135fc869fa2e. Report an issue: GitHub.