{"record":{"id":"6639ce904e40655e","repo":"quickwit-oss/tantivy","slug":"doc-store-block-not-completely-decompressed-data","errorCode":null,"errorMessage":"doc store block not completely decompressed, data corruption","messagePattern":"doc store block not completely decompressed, data corruption","errorType":"exception","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"src/store/compression_lz4_block.rs","lineNumber":43,"sourceCode":"\n#[inline]\n#[expect(clippy::uninit_vec)]\npub fn decompress(compressed: &[u8], decompressed: &mut Vec<u8>) -> io::Result<()> {\n    decompressed.clear();\n    let uncompressed_size_bytes: &[u8; 4] = compressed\n        .get(..4)\n        .ok_or(io::ErrorKind::InvalidData)?\n        .try_into()\n        .unwrap();\n    let uncompressed_size = u32::from_le_bytes(*uncompressed_size_bytes) as usize;\n    decompressed.reserve(uncompressed_size);\n    unsafe {\n        decompressed.set_len(uncompressed_size);\n    }\n    let bytes_written = decompress_into(&compressed[4..], decompressed)\n        .map_err(|err| io::Error::new(io::ErrorKind::InvalidData, err.to_string()))?;\n    if bytes_written != uncompressed_size {\n        return Err(io::Error::new(\n            io::ErrorKind::InvalidData,\n            \"doc store block not completely decompressed, data corruption\".to_string(),\n        ));\n    }\n    Ok(())\n}\n","sourceCodeStart":25,"sourceCodeEnd":50,"githubUrl":"https://github.com/quickwit-oss/tantivy/blob/b5d8deb80c26924e6b007a5b1a7630f35ca64de4/src/store/compression_lz4_block.rs#L25-L50","documentation":"Thrown by decompress() when lz4 decompression succeeds but produces fewer bytes than the u32 uncompressed length recorded in the block's 4-byte header. This mismatch indicates the block is incompletely written or corrupted, so the library refuses to return partial data.","triggerScenarios":"Reading a doc store block where bytes_written from decompress_into differs from uncompressed_size parsed from the block header; typically a truncated or partially overwritten segment file.","commonSituations":"Crash or kill during index write leaving a partially flushed block; disk corruption/bit rot on segment files; incomplete file transfer of index directories.","solutions":["Restore the index from backup or rebuild it by re-indexing the source documents","Check whether the writing process crashed mid-flush and regenerate the affected segment","Validate file sizes/checksums against the manifest to locate the corrupted segment","Ensure complete, atomic copying of index directories (no partial rsync/scp)"],"exampleFix":"// before\nif bytes_written != uncompressed_size {\n    return Err(io::Error::new(io::ErrorKind::InvalidData, \"doc store block not completely decompressed, data corruption\".to_string()));\n}\n// after\nif bytes_written != uncompressed_size {\n    return Err(io::Error::new(io::ErrorKind::InvalidData, format!(\"doc store block not completely decompressed: expected {uncompressed_size}, got {bytes_written}\")));\n}","handlingStrategy":"try-catch","validationCode":"fn block_header_matches(compressed: &[u8], expected: Option<usize>) -> bool {\n    if compressed.len() < 4 { return false; }\n    let declared = u32::from_le_bytes(compressed[0..4].try_into().unwrap()) as usize;\n    match expected {\n        Some(n) => declared == n,\n        None => declared > 0,\n    }\n}","typeGuard":"fn has_plausible_block_header(buf: &[u8]) -> bool {\n    buf.len() >= 4 && {\n        let declared = u32::from_le_bytes(buf[0..4].try_into().unwrap()) as usize;\n        declared > 0 && declared < 1 << 30\n    }\n}","tryCatchPattern":"match decompress(&block, &mut buf) {\n    Err(e) if e.to_string().contains(\"not completely decompressed\") => {\n        // corruption guard tripped: quarantine segment and re-index\n        quarantine_segment();\n    }\n    Err(e) => return Err(e),\n    Ok(()) => {},\n}","preventionTips":["Ensure index writes complete atomically (write to temp + rename) before readers open segments","Shut down writers gracefully; avoid SIGKILL during commits","Verify segment file sizes against the manifest after any copy/move","Restore from backup rather than retrying reads on corrupted blocks"],"tags":["io","data-corruption","truncation","rust"],"backgroundTag":"doc-store-block-truncated","analyzedSha":"b5d8deb80c26924e6b007a5b1a7630f35ca64de4","analyzedAt":"2026-09-05T13:20:51.521Z","contentChangedAt":"2026-09-05T13:20:51.521Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}