quickwit-oss/tantivy · error · io::Error

InvalidData

InvalidData

Error message

Could not convert value to compact_space. This is a bug.

What it means

During u128 fast-field compression, each value is mapped into a compact space defined by the codec params; u128_to_compact is total over the space the params were computed for. If any value falls outside it, this io::Error (InvalidData) is returned because it indicates an internal invariant violation — hence the message "This is a bug". Users normally see it only via corrupted metadata or a genuine library bug.

Source

Thrown at columnar/src/column_values/u128_based/compact_space/mod.rs:241

        let footer_len = writer.written_bytes() as u32;
        footer_len.serialize(writer)?;

        Ok(())
    }

    pub fn compress_into(
        self,
        vals: impl Iterator<Item = u128>,
        write: &mut impl Write,
    ) -> io::Result<()> {
        let mut bitpacker = BitPacker::default();
        for val in vals {
            let compact = self
                .params
                .compact_space
                .u128_to_compact(val)
                .map_err(|_| {
                    io::Error::new(
                        io::ErrorKind::InvalidData,
                        "Could not convert value to compact_space. This is a bug.",
                    )
                })?;
            bitpacker.write(compact as u64, self.params.num_bits, write)?;
        }
        bitpacker.close(write)?;
        self.write_footer(write)?;
        Ok(())
    }
}

#[derive(Debug, Clone)]
pub struct CompactSpaceDecompressor {
    data: OwnedBytes,
    params: IPCodecParams,
}

View on GitHub (pinned to b5d8deb80c)

Solutions

  1. Report to tantivy with a reproducer — the message explicitly indicates a library bug.
  2. Re-index the affected segment so params and values are recomputed together.
  3. Verify no external tool rewrote or truncated columnar files.
  4. Upgrade tantivy in case the bug is already fixed.

Example fix

// before
let compact = params.compact_space.u128_to_compact(val)?; // may fail
// after
// ensure params are derived from the exact same column values:
let params = CompactSpaceParams::from_values(vals)?;
let compact = params.compact_space.u128_to_compact(val)?; // total by construction
Defensive patterns

Strategy: try-catch

Try / catch

let compact = match params.compact_space.u128_to_compact(val) {
    Ok(c) => c,
    Err(_) => {
        // "This is a bug": log value + params, rebuild segment, report upstream
        unreachable_rebuild_segment();
    }
};

Prevention

When it happens

Trigger: Calling compress_into (directly or via serialize_column_values_u128/compress) where params' compact space was computed from different values than those being compressed — e.g. mismatched stats, corrupted header describing value range, or memory corruption.

Common situations: Segment files whose serialized codec params don't match the values (truncation/partial rewrite); concurrency bugs during serialization; tantivy internal bugs (report upstream).

Related errors

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/19dfc6acabdb6c04. Report an issue: GitHub.