quickwit-oss/tantivy · error

Corrupted data. Invalid VInt 32

Error message

Corrupted data. Invalid VInt 32

What it means

VInt (variable-length integer) encoding terminates each byte with a stop bit. vint_len scans up to 5 bytes looking for the terminator; if none is found within 5 bytes the value cannot be a valid 32-bit vint, so the data is corrupt. This indicates deserialization from a truncated or corrupted buffer.

Source

Thrown at common/src/vint.rs:129

    &buf[0..num_bytes]
}

/// Returns the number of bytes covered by a
/// serialized vint `u32`.
///
/// Expects a buffer data that starts
/// by the serialized `vint`, scans at most 5 bytes ahead until
/// it finds the vint final byte.
///
/// # May Panic
/// If the payload does not start by a valid `vint`
fn vint_len(data: &[u8]) -> usize {
    for (i, &val) in data.iter().enumerate().take(5) {
        if val >= STOP_BIT {
            return i + 1;
        }
    }
    panic!("Corrupted data. Invalid VInt 32");
}

/// Reads a vint `u32` from a buffer, and
/// consumes its payload data.
///
/// # Panics
///
/// If the buffer does not start by a valid
/// vint payload
pub fn read_u32_vint(data: &mut &[u8]) -> u32 {
    let (result, vlen) = read_u32_vint_no_advance(data);
    *data = &data[vlen..];
    result
}

pub fn read_u32_vint_no_advance(data: &[u8]) -> (u32, usize) {
    let vlen = vint_len(data);
    let mut result = 0u32;

View on GitHub (pinned to b5d8deb80c)

Solutions

1. Validate and re-download / restore the corrupted segment files, or rebuild the index from source documents.
2. Check that the reader's offset is correct — a prior field misparse can leave the cursor mid-vint; verify preceding read lengths.
3. Confirm the file was written by a compatible tantivy version (vint format mismatches across versions).

Example fix

// before
let (val, len) = read_u32_vint_no_advance(data, offset)?; // panics on 5 continuation bytes
// after
if data.len() < offset + 5 {
    return Err(corruption_error("truncated vint"));
}
let (val, len) = read_u32_vint_no_advance(data, offset)?;

if data.len() < offset + 5 {
    return Err(corruption());
}

// panic is unrecoverable; isolate parsing
let result = std::panic::catch_unwind(|| read_vint(data, offset));
match result {
    Ok(v) => v,
    Err(_) => { mark_segment_corrupted(seg); skip_segment(seg) }
}

When it happens

Trigger: Calling read_u32_vint_no_advance (or any vint deserialization) on a byte slice where the first 5 bytes all have the continuation (stop) bit set — truncated input, misaligned buffer offset, or reading past the logical end of the vint payload.

Common situations: Corrupted/truncated index files (incomplete download, disk failure, partial write); reading at a wrong offset after a prior parsing bug; opening a segment written by a different/incompatible version; mixing files between index versions.

Related errors

• UnexpectedEof
• InvalidData
• InvalidData
• InvalidData
• InvalidData

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/d6fed8cc49ef4a4a. Report an issue: GitHub.