quickwit-oss/quickwit · error

storekey encode prefix-length sentinel: {}

Error message

storekey encode prefix-length sentinel: {}

What it means

After encoding all prefix column values, the code appends a u8 sentinel (the prefix length) via storekey::encode to make the per-RG key a byte-for-byte prefix of each row's sorted_series key. This error wraps any failure of that storekey encoding. Since encoding a plain u8 cannot realistically fail, this almost always signals an internal bug or a corrupted/unsupported storekey version.

Source

Thrown at quickwit/quickwit-parquet-engine/src/merge/streaming/region_grouping.rs:295

        encode_prefix_col_value(stats, col, rg_idx, input_idx, &mut key)?;
    }

    // Trailing prefix-length sentinel: an additional `u8(prefix_len)`
    // ordinal byte that does two things at once:
    //
    // 1. **Forces nulls-last ordering across RGs.** For prefix_len=1 an all-null RG produces an
    //    empty per-column body and would otherwise lex-sort *before* any non-null RG. With the
    //    sentinel, the all-null key becomes `[prefix_len]` and the non-null key becomes `[ord(0),
    //    storekey(value), ..., prefix_len]`. The non-null key starts with `ord(0) = 0x00`, smaller
    //    than `prefix_len >= 1`, so non-null RGs sort first — matching `sorted_series`'s row-level
    //    nulls-last convention via the same "the next ordinal byte appears in the skipped slot"
    //    mechanism.
    // 2. **Preserves the "literal prefix of sorted_series" property.** The byte we append is
    //    exactly what `sorted_series` writes right after the prefix columns: the ordinal of the
    //    next sort-schema column (`u8(prefix_len)`). So the per-RG key remains a byte-for-byte
    //    prefix of every row's `sorted_series` value in that RG.
    storekey::encode(&mut key, &(prefix_cols.len() as u8))
        .map_err(|e| anyhow!("storekey encode prefix-length sentinel: {}", e))?;

    Ok(key)
}

/// Verify `min == max` on the column chunk's non-null stats and
/// append the single value to `key` via
/// [`crate::sorted_series::append_prefix_col_to_key`] (which handles
/// the ordinal prefix + descending-direction byte inversion). Caller
/// has already filtered out all-null and mixed-null cases.
///
/// `Statistics::ByteArray` values are routed through the
/// `Encode for str` impl after a UTF-8 check — every realistic sort
/// prefix column (`metric_name`, `service`, tag names) is UTF-8
/// text, and `sorted_series` itself only encodes strings, so the
/// "byte prefix of sorted_series" property only holds for UTF-8
/// values. Non-UTF-8 byte-array prefix cols would never match a
/// `sorted_series` key in practice (sorted_series would not encode
/// them either) and so are rejected up front.

View on GitHub (pinned to a39730c5cd)

Solutions

1. Inspect the storekey error payload in the message; if it indicates capacity or version issues, update/fix the storekey crate usage.
2. Audit recent changes to storekey::encode for u8/i-ordinal types.
3. If reproducible, add a unit test asserting storekey::encode of a u8 succeeds to catch regressions early.

if let Err(e) = extract_rg_composite_prefix_key(...) {
    if e.to_string().contains("storekey encode prefix-length sentinel") {
        // this is an internal invariant break: capture context and report as a bug
        log::error!("storekey sentinel encode failed — report with full error: {e:#}");
    }
    return Err(e);
}

When it happens

Trigger: extract_rg_composite_prefix_key reaching the sentinel encode step and storekey::encode returning an Err for the u8 prefix-length value — effectively only on a storekey internal failure.

Common situations: A storekey version/impl change in this workspace; an unexpected buffer limitation; genuinely a can't-happen path indicating an invariant break.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• timestamp_secs must be UInt64 or Int64 for MC-3 check
• input {input_idx} rg {rg_idx} col '{}' has non-UTF-8 byte-ar
• node not found in pending
• OTP logs or traces do not support VRL transforms
• position of a Kafka partition should never be EOF

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/57e6efcc977e7789. Report an issue: GitHub.