quickwit-oss/tantivy · error · io::Error

doc store block not completely decompressed, data corruption

Error message

doc store block not completely decompressed, data corruption

What it means

Thrown by the zstd-based decompress() when zstd decompression returns a size that differs from the uncompressed_size stored in the block header. Like the lz4 variant, this is a deliberate data-corruption guard: the library will not hand back a partially decompressed block.

Source

Thrown at src/store/compression_zstd_block.rs:47

#[inline]
pub fn decompress(compressed: &[u8], decompressed: &mut Vec<u8>) -> io::Result<()> {
    let count_size = std::mem::size_of::<u32>();
    let uncompressed_size = u32::from_le_bytes(
        compressed
            .get(..count_size)
            .ok_or(io::ErrorKind::InvalidData)?
            .try_into()
            .unwrap(),
    ) as usize;

    decompressed.clear();
    decompressed.resize(uncompressed_size, 0);

    let decompressed_size = decompress_to_buffer(&compressed[count_size..], decompressed)?;

    if decompressed_size != uncompressed_size {
        return Err(io::Error::new(
            io::ErrorKind::InvalidData,
            "doc store block not completely decompressed, data corruption".to_string(),
        ));
    }

    Ok(())
}

View on GitHub (pinned to b5d8deb80c)

Solutions

Rebuild the index from source documents or restore from a verified backup
Verify segment file integrity and re-transfer index files completely
Match the tantivy (and compression feature) versions used for writing and reading
Delete and regenerate the corrupted segment instead of retrying reads

Example fix

// before
if decompressed_size != uncompressed_size {
    return Err(io::Error::new(io::ErrorKind::InvalidData, "doc store block not completely decompressed, data corruption".to_string()));
}
// after
if decompressed_size != uncompressed_size {
    return Err(io::Error::new(io::ErrorKind::InvalidData, format!("zstd block incomplete: expected {uncompressed_size}, got {decompressed_size}")));
}

Defensive patterns

Strategy: try-catch

Validate before calling

fn validate_zstd_block_header(compressed: &[u8]) -> io::Result<usize> {
    if compressed.len() < 5 {
        return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "block too short for size header + payload"));
    }
    // first 4 bytes carry the count/size prefix
    Ok(u32::from_le_bytes(compressed[0..4].try_into().unwrap()) as usize)
}

Type guard

fn plausible_uncompressed_size(buf: &[u8]) -> bool {
    buf.len() > 4 && u32::from_le_bytes(buf[0..4].try_into().unwrap()) > 0
}

Try / catch

match decompress(&block, &mut buf) {
    Err(e) if e.kind() == io::ErrorKind::InvalidData && e.to_string().contains("not completely decompressed") => {
        return Err(IndexError::CorruptedBlock);
    }
    Err(e) => return Err(e.into()),
    Ok(()) => {},
}

Prevention

Keep zstd compression feature flags consistent between writer and reader builds
Verify file integrity (checksums) after transfers and before opening
Avoid partial writes: finalize segment files before making them visible to readers
Rebuild corrupted segments instead of retrying decompression

When it happens

Trigger: Reading a zstd-compressed doc store block where decompress_to_buffer yields decompressed_size != uncompressed_size; corrupted or truncated segment payloads.

Common situations: Index segments damaged on disk; blocks written by an incompatible tantivy/zstd configuration; interrupted index writes or incomplete file copies.

Related errors

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/6d8cd8d93c413c90. Report an issue: GitHub.