{"record":{"id":"6d8cd8d93c413c90","repo":"quickwit-oss/tantivy","slug":"doc-store-block-not-completely-decompressed-data-6d8cd8","errorCode":null,"errorMessage":"doc store block not completely decompressed, data corruption","messagePattern":"doc store block not completely decompressed, data corruption","errorType":"exception","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"src/store/compression_zstd_block.rs","lineNumber":47,"sourceCode":"\n#[inline]\npub fn decompress(compressed: &[u8], decompressed: &mut Vec<u8>) -> io::Result<()> {\n    let count_size = std::mem::size_of::<u32>();\n    let uncompressed_size = u32::from_le_bytes(\n        compressed\n            .get(..count_size)\n            .ok_or(io::ErrorKind::InvalidData)?\n            .try_into()\n            .unwrap(),\n    ) as usize;\n\n    decompressed.clear();\n    decompressed.resize(uncompressed_size, 0);\n\n    let decompressed_size = decompress_to_buffer(&compressed[count_size..], decompressed)?;\n\n    if decompressed_size != uncompressed_size {\n        return Err(io::Error::new(\n            io::ErrorKind::InvalidData,\n            \"doc store block not completely decompressed, data corruption\".to_string(),\n        ));\n    }\n\n    Ok(())\n}\n","sourceCodeStart":29,"sourceCodeEnd":55,"githubUrl":"https://github.com/quickwit-oss/tantivy/blob/b5d8deb80c26924e6b007a5b1a7630f35ca64de4/src/store/compression_zstd_block.rs#L29-L55","documentation":"Thrown by the zstd-based decompress() when zstd decompression returns a size that differs from the uncompressed_size stored in the block header. Like the lz4 variant, this is a deliberate data-corruption guard: the library will not hand back a partially decompressed block.","triggerScenarios":"Reading a zstd-compressed doc store block where decompress_to_buffer yields decompressed_size != uncompressed_size; corrupted or truncated segment payloads.","commonSituations":"Index segments damaged on disk; blocks written by an incompatible tantivy/zstd configuration; interrupted index writes or incomplete file copies.","solutions":["Rebuild the index from source documents or restore from a verified backup","Verify segment file integrity and re-transfer index files completely","Match the tantivy (and compression feature) versions used for writing and reading","Delete and regenerate the corrupted segment instead of retrying reads"],"exampleFix":"// before\nif decompressed_size != uncompressed_size {\n    return Err(io::Error::new(io::ErrorKind::InvalidData, \"doc store block not completely decompressed, data corruption\".to_string()));\n}\n// after\nif decompressed_size != uncompressed_size {\n    return Err(io::Error::new(io::ErrorKind::InvalidData, format!(\"zstd block incomplete: expected {uncompressed_size}, got {decompressed_size}\")));\n}","handlingStrategy":"try-catch","validationCode":"fn validate_zstd_block_header(compressed: &[u8]) -> io::Result<usize> {\n    if compressed.len() < 5 {\n        return Err(io::Error::new(io::ErrorKind::UnexpectedEof, \"block too short for size header + payload\"));\n    }\n    // first 4 bytes carry the count/size prefix\n    Ok(u32::from_le_bytes(compressed[0..4].try_into().unwrap()) as usize)\n}","typeGuard":"fn plausible_uncompressed_size(buf: &[u8]) -> bool {\n    buf.len() > 4 && u32::from_le_bytes(buf[0..4].try_into().unwrap()) > 0\n}","tryCatchPattern":"match decompress(&block, &mut buf) {\n    Err(e) if e.kind() == io::ErrorKind::InvalidData && e.to_string().contains(\"not completely decompressed\") => {\n        return Err(IndexError::CorruptedBlock);\n    }\n    Err(e) => return Err(e.into()),\n    Ok(()) => {},\n}","preventionTips":["Keep zstd compression feature flags consistent between writer and reader builds","Verify file integrity (checksums) after transfers and before opening","Avoid partial writes: finalize segment files before making them visible to readers","Rebuild corrupted segments instead of retrying decompression"],"tags":["io","compression","zstd","data-corruption","rust"],"backgroundTag":"doc-store-block-truncated","analyzedSha":"b5d8deb80c26924e6b007a5b1a7630f35ca64de4","analyzedAt":"2026-09-05T13:20:51.521Z","contentChangedAt":"2026-09-05T13:20:51.521Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}