quickwit-oss/tantivy · error

Invalid op metadata byte

Error message

Invalid op metadata byte

What it means

ColumnOperation::deserialize parses an in-memory serialized column operation; the first byte is a metadata code that ColumnOperationMetadata::try_from_code must recognize. An unknown/invalid code hits .expect("Invalid op metadata byte") and panics. Because deserialize targets in-memory buffers produced by the same code, an unrecognized byte means memory corruption, a desynced read position, or version skew between writer and reader of the operation format.

Source

Thrown at columnar/src/columnar/writer/column_operation.rs:98

                    len: symbol_len,
                }
            }
        };
        minibuf.bytes[0] = column_op_metadata.to_code();
        // +1 for the metadata
        minibuf.len = 1 + column_op_metadata.len;
        minibuf
    }

    /// Deserialize a column operation.
    /// Returns None if the buffer is empty.
    ///
    /// Panics if the payload is invalid:
    /// this deserialize method is meant to target in memory.
    pub(super) fn deserialize(bytes: &mut &[u8]) -> Option<Self> {
        let column_op_metadata_byte = pop_first_byte(bytes)?;
        let column_op_metadata = ColumnOperationMetadata::try_from_code(column_op_metadata_byte)
            .expect("Invalid op metadata byte");
        let symbol_bytes: &[u8];
        (symbol_bytes, *bytes) = bytes.split_at(column_op_metadata.len as usize);
        match column_op_metadata.op_type {
            ColumnOperationType::NewDoc => {
                let new_doc = u32::deserialize(symbol_bytes);
                Some(ColumnOperation::NewDoc(new_doc))
            }
            ColumnOperationType::AddValue => {
                let value = V::deserialize(symbol_bytes);
                Some(ColumnOperation::Value(value))
            }
        }
    }
}

impl<T> From<T> for ColumnOperation<T> {
    fn from(value: T) -> Self {
        ColumnOperation::Value(value)

View on GitHub (pinned to b5d8deb80c)

Solutions

1. Verify the read cursor is exactly at an operation boundary; re-check preceding split_at/length arithmetic.
2. Ensure writer and reader code versions agree on ColumnOperationType/ColumnOperationMetadata codes.
3. Replace the expect with Option propagation (return None) if the format can legitimately contain unknown codes, for forward compatibility.
4. If the data comes from disk, re-check serialization/deserialization symmetry with round-trip tests.

Example fix

// before
let column_op_metadata = ColumnOperationMetadata::try_from_code(byte).expect("Invalid op metadata byte");
// after
let column_op_metadata = ColumnOperationMetadata::try_from_code(byte).ok()?;

// Validate the byte slice begins with a known op code before deserialize
fn op_code_known(bytes: &[u8]) -> bool {
    bytes.first()
        .map(|b| ColumnOperationMetadata::try_from_code(*b).is_ok())
        .unwrap_or(false)
}

// panic-based; wrap and treat as data corruption
let op = std::panic::catch_unwind(|| ColumnOperation::deserialize(&mut bytes))
    .unwrap_or_else(|_| panic-or-error);

When it happens

Trigger: Calling deserialize on a byte slice whose first byte is not a valid ColumnOperationMetadata code: reading at a wrong offset, byte buffer truncated/overwritten, or data written by a different version with new op codes.

Common situations: Hand-rolling reads over the column operation stream and losing byte alignment; concurrent mutation of the buffer; cross-version playback of operation logs.

Related errors

• All columns re required to be numerical
• TermMissingAgg collector, but no missing found in agg req
• No multivalued index is allowed when stacking column index
• Internal Error: Called get_val of empty column.
• Corrupted data. Invalid VInt 32

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/d2ab60411235d72d. Report an issue: GitHub.