quickwit-oss/tantivy · error · io::Error

err.to_string()

Error message

err.to_string()

What it means

Thrown by compress() when lz4_flex's compress_into fails while compressing a doc store block. The underlying lz4 error is flattened into its string form and wrapped in an io::Error with ErrorKind::InvalidData. This library treats any lz4 compression failure as invalid input/state rather than retrying.

Source

Thrown at src/store/compression_lz4_block.rs:17

use std::io::{self};
use std::mem;

use lz4_flex::{compress_into, decompress_into};

#[inline]
#[expect(clippy::uninit_vec)]
pub fn compress(uncompressed: &[u8], compressed: &mut Vec<u8>) -> io::Result<()> {
    compressed.clear();
    let maximum_output_size =
        mem::size_of::<u32>() + lz4_flex::block::get_maximum_output_size(uncompressed.len());
    compressed.reserve(maximum_output_size);
    unsafe {
        compressed.set_len(maximum_output_size);
    }
    let bytes_written = compress_into(uncompressed, &mut compressed[4..])
        .map_err(|err| io::Error::new(io::ErrorKind::InvalidData, err.to_string()))?;
    let num_bytes = uncompressed.len() as u32;
    compressed[0..4].copy_from_slice(&num_bytes.to_le_bytes());
    unsafe {
        compressed.set_len(bytes_written + mem::size_of::<u32>());
    }
    Ok(())
}

#[inline]
#[expect(clippy::uninit_vec)]
pub fn decompress(compressed: &[u8], decompressed: &mut Vec<u8>) -> io::Result<()> {
    decompressed.clear();
    let uncompressed_size_bytes: &[u8; 4] = compressed
        .get(..4)
        .ok_or(io::ErrorKind::InvalidData)?
        .try_into()
        .unwrap();
    let uncompressed_size = u32::from_le_bytes(*uncompressed_size_bytes) as usize;

View on GitHub (pinned to b5d8deb80c)

Solutions

  1. Inspect the wrapped lz4 error message embedded in the io::Error to identify the exact lz4_flex failure
  2. Verify the input byte slice being compressed is well-formed and of expected size
  3. Check that lz4_flex and tantivy versions are compatible and up to date
  4. If it reproduces on valid data, report it with a minimal reproducing payload

Example fix

// before
let bytes_written = compress_into(uncompressed, &mut compressed[4..])?;
// after
let bytes_written = compress_into(uncompressed, &mut compressed[4..])
    .map_err(|err| io::Error::new(io::ErrorKind::InvalidData, format!("lz4 compression failed: {err}")))?;
Defensive patterns

Strategy: validation

Validate before calling

fn ensure_compressible(data: &[u8]) -> io::Result<()> {
    if data.is_empty() {
        return Err(io::Error::new(io::ErrorKind::InvalidInput, "cannot compress empty payload"));
    }
    let max_out = std::mem::size_of::<u32>() + lz4_flex::block::get_maximum_output_size(data.len());
    if max_out > isize::MAX as usize {
        return Err(io::Error::new(io::ErrorKind::InvalidInput, "payload too large for lz4 block compression"));
    }
    Ok(())
}

Type guard

fn is_compressible_input(data: &[u8]) -> bool {
    !data.is_empty() && data.len() <= lz4_flex::block::MAX_INPUT_SIZE
}

Try / catch

match compress(uncompressed, &mut compressed) {
    Ok(()) => {},
    Err(e) if e.kind() == io::ErrorKind::InvalidData => eprintln!("lz4 compression rejected input: {e}"),
    Err(e) => return Err(e),
}

Prevention

When it happens

Trigger: Calling compress() (directly or via doc store writer) with a payload for which lz4_flex::block::compress_into reports an error, e.g. output buffer sizing issues or lz4 internal limits; the error surfaces when writing documents into a compressed doc store block.

Common situations: Writing extremely large or pathological byte payloads into the block compressor; mismatched tantivy/lz4_flex crate versions where maximum_output_size bookkeeping no longer matches the library's expectations; corrupted in-memory buffers passed in by the caller.

Related errors

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/783a645f52e1a7e0. Report an issue: GitHub.