quickwit-oss/quickwit · error · io::Error (InvalidData)

dynamic protobuf decode error wrapped as io::Error (InvalidD

Error message

dynamic protobuf decode error wrapped as io::Error (InvalidData) via map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))

What it means

After reading the version byte, `deserialize` zstd-decompresses the remainder and decodes it as a protobuf message with prost's `Self::decode`; any DecodeError is wrapped into io::Error(InvalidData) via map_err. The error therefore indicates valid envelope framing but corrupt/truncated compressed payload or a protobuf body that does not match the schema.

Source

Thrown at quickwit/quickwit-proto/src/search/mod.rs:269

    pub fn deserialize<R: Read>(mut reader: R) -> io::Result<Self> {
        let mut version_byte = [0u8; 1];
        reader.read_exact(&mut version_byte)?;

        if version_byte[0] != FIELDS_METADATA_FORMAT_VERSION {
            return Err(io::Error::new(
                io::ErrorKind::InvalidData,
                format!(
                    "unsupported split fields format version: {}",
                    version_byte[0]
                ),
            ));
        }
        let mut zstd_decoder = zstd::stream::read::Decoder::new(reader)?;
        let mut decompressed = Vec::new();
        zstd_decoder.read_to_end(&mut decompressed)?;

        Self::decode(&decompressed[..])
            .map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))
    }
}

impl ListFieldsEntry {
    pub fn cmp_by_name_and_type(&self, other: &Self) -> Ordering {
        self.field_name
            .cmp(&other.field_name)
            .then_with(|| self.field_type.cmp(&other.field_type))
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    fn entry(field_name: &str) -> ListFieldsEntry {
        ListFieldsEntry {
            field_name: field_name.to_string(),

View on GitHub (pinned to a39730c5cd)

Solutions

Inspect the inner prost DecodeError (chained as the io error's source) to identify the failing field.
Regenerate the metadata by re-indexing or re-serializing from a known-good source.
Confirm the quickwit-proto schema used by the reader matches the writer's schema/version.
Check the storage object's integrity (ETag/checksum) against what the writer uploaded.

Defensive patterns

Strategy: try-catch

Try / catch

match SplitFieldsMetadata::deserialize(reader) {
    Err(e) if e.kind() == std::io::ErrorKind::InvalidData => {
        // check e.source(): prost DecodeError tells which field failed to decode
        regenerate_or_reindex()
    }
    other => other?,
}

Prevention

Verify object checksums/ETags after uploading split metadata.
Avoid concurrent overwrites of the same metadata key.
Keep protobuf schema changes backward-compatible or bump the version byte.

When it happens

Trigger: Calling deserialize on metadata whose zstd frame is truncated or corrupt, or whose decompressed bytes are not a valid serialized message — e.g. bytes written by a different schema version that still passes the version check, or bit-corrupted objects in storage.

Common situations: Corrupted objects after interrupted uploads; mismatched protobuf schema between writer and reader crates; storage returning wrong object content under the same key; manual payload manipulation in tooling.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/90d51382fbbf168e. Report an issue: GitHub.