quickwit-oss/tantivy · error · io::Error

InvalidData

InvalidData

Error message

Unknown code `{code}.`

What it means

U128FastFieldCodec::deserialize reads a one-byte codec type code and maps it back via from_code; an unrecognized byte yields InvalidData with this message. Note the message literally contains backticks around `{code}.` — the format string was never interpolated, so the byte value is not shown. It signals reading data written with an unknown/newer codec type or corruption.

Source

Thrown at columnar/src/column_values/u128_based/mod.rs:84

#[derive(PartialEq, Eq, PartialOrd, Ord, Debug, Clone, Copy)]
#[repr(u8)]
/// Available codecs to use to encode the u128 (via [`MonotonicallyMappableToU128`]) converted data.
pub(crate) enum U128FastFieldCodecType {
    /// This codec takes a large number space (u128) and reduces it to a compact number space, by
    /// removing the holes.
    CompactSpace = 1,
}

impl BinarySerializable for U128FastFieldCodecType {
    fn serialize<W: Write + ?Sized>(&self, wrt: &mut W) -> io::Result<()> {
        self.to_code().serialize(wrt)
    }

    fn deserialize<R: io::Read>(reader: &mut R) -> io::Result<Self> {
        let code = u8::deserialize(reader)?;
        let codec_type: Self = Self::from_code(code)
            .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidData, "Unknown code `{code}.`"))?;
        Ok(codec_type)
    }
}

impl U128FastFieldCodecType {
    pub(crate) fn to_code(self) -> u8 {
        self as u8
    }

    pub(crate) fn from_code(code: u8) -> Option<Self> {
        match code {
            1 => Some(Self::CompactSpace),
            _ => None,
        }
    }
}

/// Returns the correct codec reader wrapped in the `Arc` for the data.

View on GitHub (pinned to b5d8deb80c)

Solutions

1. Use a tantivy version at least as new as the one that wrote the index.
2. Rebuild the segment/index with the current version.
3. Check for corruption — validate the byte stream alignment before the codec code.
4. Patch from_code if adding a custom codec type so the code round-trips.

Example fix

// before
let codec = CodecType::deserialize(&mut reader)?; // InvalidData on unknown code
// after
// ensure the writer's tantivy version is supported before opening the index,
// or map the raw code yourself:
let code = u8::deserialize(&mut reader)?;
let codec = CodecType::from_code(code)
    .ok_or_else(|| format!("unsupported codec code {code}").into())?;

fn is_known_codec_code(code: u8) -> bool {
    CodecType::from_code(code).is_some()
}

match CodecType::deserialize(&mut reader) {
    Err(e) if e.kind() == std::io::ErrorKind::InvalidData
        && e.to_string().starts_with("Unknown code") => {
        // index written by newer tantivy or corrupt: upgrade reader or rebuild
    }
    other => other?,
}

When it happens

Trigger: Deserializing a u128 fast field whose codec code byte is not among the known from_code values — e.g. index written by a newer tantivy with a new codec, byte misalignment, or corrupted files.

Common situations: Cross-version index reads (new writer, old reader); damaged segment files; manually edited or partially written columnar data.

Related errors

• Corrupted data. Invalid VInt 32
• UnexpectedEof
• InvalidData
• InvalidData
• InvalidData

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/f4a605cbf37321ab. Report an issue: GitHub.