What causes "Reach end of buffer while reading VInt" in quickwit-oss/tantivy?

Validate the byte slice length against the expected varint/record boundaries before deserializing

quickwit-oss/tantivy · error · io::Error

InvalidData

InvalidData

Error message

Reach end of buffer while reading VInt

What it means

VInt/U128 varint deserialization reads bytes until one has the STOP_BIT set. If the buffer ends before a stop byte is found, deserialize raises InvalidData 'Reach end of buffer while reading VInt' — the varint is truncated. This indicates truncation or a misaligned read position.

Source

Thrown at common/src/vint.rs:47

    }

    #[allow(clippy::unbuffered_bytes)]
    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self> {
        #[allow(clippy::unbuffered_bytes)]
        let mut bytes = reader.bytes();
        let mut result = 0u128;
        let mut shift = 0u64;
        loop {
            match bytes.next() {
                Some(Ok(b)) => {
                    result |= u128::from(b % 128u8) << shift;
                    if b >= STOP_BIT {
                        return Ok(VIntU128(result));
                    }
                    shift += 7;
                }
                _ => {
                    return Err(io::Error::new(
                        io::ErrorKind::InvalidData,
                        "Reach end of buffer while reading VInt",
                    ));
                }
            }
        }
    }
}

///   Wrapper over a `u64` that serializes as a variable int.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub struct VInt(pub u64);

const STOP_BIT: u8 = 128;

#[inline]
pub fn serialize_vint_u32(val: u32, buf: &mut [u8; 8]) -> &[u8] {
    const START_2: u64 = 1 << 7;

View on GitHub (pinned to b5d8deb80c)

Solutions

Validate the byte slice length against the expected varint/record boundaries before deserializing
Verify file integrity (checksums/footers) to catch truncation
Check that read offsets are correctly aligned with preceding serialized fields
Re-obtain or re-index the corrupted data

Example fix

// before
let v = VIntU128::deserialize(&mut bytes)?;
// after
if bytes.as_slice().is_empty() { return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "no bytes for VInt")); }
let v = VIntU128::deserialize(&mut bytes)?;

Defensive patterns

Strategy: validation

Validate before calling

fn has_room_for_varint(bytes: &[u8]) -> bool {
    // ensure at least one remaining byte; full validation is walking until STOP_BIT
    !bytes.is_empty()
}
// better: walk: for (i, b) in bytes.iter().enumerate() { if b >= 128 { /* ok */ } }

Try / catch

match VIntU128::deserialize(&mut bytes.clone()) {
    Err(e) if e.kind() == io::ErrorKind::InvalidData && e.to_string().contains("Reach end of buffer while reading VInt") => {
        // truncated buffer: verify file length/checksum or fix read offset
    }
    other => other?,
}

Prevention

Validate buffers contain a STOP_BIT-terminated varint before reading
Verify file checksums/footers to catch truncation early
Recheck offset arithmetic for preceding variable-length fields
Persist and compare expected lengths at write time to detect partial writes

When it happens

Trigger: deserialize (read_u64/unsafe_read_u64-style varint readers in vint.rs) consuming a byte slice that ends while the high bit of the current byte is still clear — truncated buffer, wrong read offset, or corrupt length prefixes.

Common situations: Truncated segment/footer files from incomplete writes or downloads; reading a varint at an offset one byte off (misalignment makes the terminator fall outside the slice); corrupt length-prefixed blocks.

Related errors

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/ec7702c42fecf825. Report an issue: GitHub.