quickwit-oss/tantivy · error · io::Error

doc store block not completely decompressed, data corruption

Error message

doc store block not completely decompressed, data corruption

What it means

Thrown by decompress() when lz4 decompression succeeds but produces fewer bytes than the u32 uncompressed length recorded in the block's 4-byte header. This mismatch indicates the block is incompletely written or corrupted, so the library refuses to return partial data.

Source

Thrown at src/store/compression_lz4_block.rs:43

#[inline]
#[expect(clippy::uninit_vec)]
pub fn decompress(compressed: &[u8], decompressed: &mut Vec<u8>) -> io::Result<()> {
    decompressed.clear();
    let uncompressed_size_bytes: &[u8; 4] = compressed
        .get(..4)
        .ok_or(io::ErrorKind::InvalidData)?
        .try_into()
        .unwrap();
    let uncompressed_size = u32::from_le_bytes(*uncompressed_size_bytes) as usize;
    decompressed.reserve(uncompressed_size);
    unsafe {
        decompressed.set_len(uncompressed_size);
    }
    let bytes_written = decompress_into(&compressed[4..], decompressed)
        .map_err(|err| io::Error::new(io::ErrorKind::InvalidData, err.to_string()))?;
    if bytes_written != uncompressed_size {
        return Err(io::Error::new(
            io::ErrorKind::InvalidData,
            "doc store block not completely decompressed, data corruption".to_string(),
        ));
    }
    Ok(())
}

View on GitHub (pinned to b5d8deb80c)

Solutions

Restore the index from backup or rebuild it by re-indexing the source documents
Check whether the writing process crashed mid-flush and regenerate the affected segment
Validate file sizes/checksums against the manifest to locate the corrupted segment
Ensure complete, atomic copying of index directories (no partial rsync/scp)

Example fix

// before
if bytes_written != uncompressed_size {
    return Err(io::Error::new(io::ErrorKind::InvalidData, "doc store block not completely decompressed, data corruption".to_string()));
}
// after
if bytes_written != uncompressed_size {
    return Err(io::Error::new(io::ErrorKind::InvalidData, format!("doc store block not completely decompressed: expected {uncompressed_size}, got {bytes_written}")));
}

Defensive patterns

Strategy: try-catch

Validate before calling

fn block_header_matches(compressed: &[u8], expected: Option<usize>) -> bool {
    if compressed.len() < 4 { return false; }
    let declared = u32::from_le_bytes(compressed[0..4].try_into().unwrap()) as usize;
    match expected {
        Some(n) => declared == n,
        None => declared > 0,
    }
}

Type guard

fn has_plausible_block_header(buf: &[u8]) -> bool {
    buf.len() >= 4 && {
        let declared = u32::from_le_bytes(buf[0..4].try_into().unwrap()) as usize;
        declared > 0 && declared < 1 << 30
    }
}

Try / catch

match decompress(&block, &mut buf) {
    Err(e) if e.to_string().contains("not completely decompressed") => {
        // corruption guard tripped: quarantine segment and re-index
        quarantine_segment();
    }
    Err(e) => return Err(e),
    Ok(()) => {},
}

Prevention

Ensure index writes complete atomically (write to temp + rename) before readers open segments
Shut down writers gracefully; avoid SIGKILL during commits
Verify segment file sizes against the manifest after any copy/move
Restore from backup rather than retrying reads on corrupted blocks

When it happens

Trigger: Reading a doc store block where bytes_written from decompress_into differs from uncompressed_size parsed from the block header; typically a truncated or partially overwritten segment file.

Common situations: Crash or kill during index write leaving a partially flushed block; disk corruption/bit rot on segment files; incomplete file transfer of index directories.

Related errors

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/6639ce904e40655e. Report an issue: GitHub.