quickwit-oss/tantivy · error · io::Error

UnexpectedEof

UnexpectedEof

Error message

Failed to deserialize column index. Empty buffer.

What it means

open_column_index deserializes a serialized column index; the first byte encodes the cardinality. An empty buffer cannot even supply that byte, so the function returns UnexpectedEof with this message. It typically means truncated or missing columnar data.

Source

Thrown at columnar/src/column_index/serialize.rs:72

        SerializableColumnIndex::Optional(SerializableOptionalIndex {
            non_null_row_ids,
            num_rows,
        }) => serialize_optional_index(non_null_row_ids.as_ref(), num_rows, &mut output)?,
        SerializableColumnIndex::Multivalued(multivalued_index) => {
            serialize_multivalued_index(&multivalued_index, &mut output)?
        }
    }
    let column_index_num_bytes = output.written_bytes() as u32;
    Ok(column_index_num_bytes)
}

/// Open a serialized column index.
pub fn open_column_index(
    mut bytes: OwnedBytes,
    format_version: Version,
) -> io::Result<ColumnIndex> {
    if bytes.is_empty() {
        return Err(io::Error::new(
            io::ErrorKind::UnexpectedEof,
            "Failed to deserialize column index. Empty buffer.",
        ));
    }
    let cardinality_code = bytes[0];
    let cardinality = Cardinality::try_from_code(cardinality_code)?;
    bytes.advance(1);
    match cardinality {
        Cardinality::Full => Ok(ColumnIndex::Full),
        Cardinality::Optional => {
            let optional_index = super::optional_index::open_optional_index(bytes)?;
            Ok(ColumnIndex::Optional(optional_index))
        }
        Cardinality::Multivalued => {
            let multivalue_index =
                super::multivalued_index::open_multivalued_index(bytes, format_version)?;
            Ok(ColumnIndex::Multivalued(multivalue_index))
        }

View on GitHub (pinned to b5d8deb80c)

Solutions

1. Check that the segment file is complete and not truncated; re-copy or re-index.
2. Verify offsets/lengths used to slice the column index bytes come from the correct footer/meta.
3. Handle empty buffers before calling: treat as missing column and fall back to a non-indexed access path.
4. Rebuild the index if corruption is confirmed.

Example fix

// before
let idx = open_column_index(bytes, version)?; // UnexpectedEof if empty
// after
if bytes.is_empty() {
    return Ok(ColumnIndex::empty()); // or skip column
}
let idx = open_column_index(bytes, version)?;

fn open_column_index_safe(bytes: OwnedBytes, version: Version) -> io::Result<Option<ColumnIndex>> {
    if bytes.is_empty() {
        return Ok(None); // column index absent
    }
    open_column_index(bytes, version).map(Some)
}

match open_column_index(bytes, version) {
    Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => {
        // fall back to unindexed column access or rebuild segment
    }
    other => other.map(Some),
}

When it happens

Trigger: Passing an empty OwnedBytes to open_column_index — e.g. reading a zero-length column-index region from a truncated segment, a failed mmap, or wrong offset/length when loading columnar files.

Common situations: Incomplete segment writes (crash during commit), corrupted downloads/copies of index directories, misaligned byte-range reads, opening columns that were never written.

Related errors

• Corrupted data. Invalid VInt 32
• InvalidData
• InvalidData
• InvalidData
• UnexpectedEof

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/ea7a5b9b357960fc. Report an issue: GitHub.