quickwit-oss/tantivy · error · io::Error

InvalidData

InvalidData

Error message

Invalid data

What it means

columnar defines a unit struct InvalidData whose From<InvalidData> for io::Error builds an io::Error(InvalidData, "Invalid data"). Code in the columnar crate uses `?` with this marker to convert internal decode failures (e.g. failed column deserialization) into a generic InvalidData io error. The message is deliberately terse; the real cause is wherever the InvalidData marker was constructed.

Source

Thrown at columnar/src/lib.rs:71

pub type DocId = u32;

#[derive(Clone, Copy, Debug)]
pub struct RowAddr {
    pub segment_ord: u32,
    pub row_id: RowId,
}

pub use sstable::{Dictionary, TermOrdHit};
pub type Streamer<'a> = sstable::Streamer<'a, VoidSSTable>;

pub use common::DateTime;

#[derive(Copy, Clone, Debug)]
pub struct InvalidData;

impl From<InvalidData> for io::Error {
    fn from(_: InvalidData) -> Self {
        io::Error::new(io::ErrorKind::InvalidData, "Invalid data")
    }
}

/// Enum describing the number of values that can exist per document
/// (or per row if you will).
///
/// The cardinality must fit on 2 bits.
#[derive(Clone, Copy, Hash, Default, Debug, PartialEq, Eq, PartialOrd, Ord)]
#[repr(u8)]
pub enum Cardinality {
    /// All documents contain exactly one value.
    /// `Full` is the default for auto-detecting the Cardinality, since it is the most strict.
    #[default]
    Full = 0,
    /// All documents contain at most one value.
    Optional = 1,
    /// All documents may contain any number of values.
    Multivalued = 2,

View on GitHub (pinned to b5d8deb80c)

Solutions

  1. Validate index file integrity (checksums) before opening columns
  2. Ensure reader and writer use compatible tantivy versions
  3. Re-index the affected segment
  4. Locate the exact failure by catching and logging the io::Error source chain / adding context at call sites
Defensive patterns

Strategy: try-catch

Validate before calling

fn file_seems_intact(path: &std::path::Path, expected_len: u64) -> bool {
    std::fs::metadata(path).map(|m| m.len() == expected_len).unwrap_or(false)
}

Try / catch

match open_column(bytes) {
    Err(e) if e.kind() == io::ErrorKind::InvalidData && e.to_string() == "Invalid data" => {
        // treat as corrupt/incompatible segment: verify checksum, reindex or skip
    }
    other => other?,
}

Prevention

When it happens

Trigger: Any columnar deserialization path that converts a decode failure via `InvalidData.into()` — e.g. opening a column whose payload doesn't match the expected format (corrupt segment, version mismatch, wrong slice).

Common situations: Reading index files written by a different tantivy version; truncated/corrupt segment data; passing non-column bytes to a column-opening API.

Related errors

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/4350c8c6b2644890. Report an issue: GitHub.