{"record":{"id":"57e6efcc977e7789","repo":"quickwit-oss/quickwit","slug":"storekey-encode-prefix-length-sentinel","errorCode":null,"errorMessage":"storekey encode prefix-length sentinel: {}","messagePattern":"storekey encode prefix-length sentinel: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-parquet-engine/src/merge/streaming/region_grouping.rs","lineNumber":295,"sourceCode":"        encode_prefix_col_value(stats, col, rg_idx, input_idx, &mut key)?;\n    }\n\n    // Trailing prefix-length sentinel: an additional `u8(prefix_len)`\n    // ordinal byte that does two things at once:\n    //\n    // 1. **Forces nulls-last ordering across RGs.** For prefix_len=1 an all-null RG produces an\n    //    empty per-column body and would otherwise lex-sort *before* any non-null RG. With the\n    //    sentinel, the all-null key becomes `[prefix_len]` and the non-null key becomes `[ord(0),\n    //    storekey(value), ..., prefix_len]`. The non-null key starts with `ord(0) = 0x00`, smaller\n    //    than `prefix_len >= 1`, so non-null RGs sort first — matching `sorted_series`'s row-level\n    //    nulls-last convention via the same \"the next ordinal byte appears in the skipped slot\"\n    //    mechanism.\n    // 2. **Preserves the \"literal prefix of sorted_series\" property.** The byte we append is\n    //    exactly what `sorted_series` writes right after the prefix columns: the ordinal of the\n    //    next sort-schema column (`u8(prefix_len)`). So the per-RG key remains a byte-for-byte\n    //    prefix of every row's `sorted_series` value in that RG.\n    storekey::encode(&mut key, &(prefix_cols.len() as u8))\n        .map_err(|e| anyhow!(\"storekey encode prefix-length sentinel: {}\", e))?;\n\n    Ok(key)\n}\n\n/// Verify `min == max` on the column chunk's non-null stats and\n/// append the single value to `key` via\n/// [`crate::sorted_series::append_prefix_col_to_key`] (which handles\n/// the ordinal prefix + descending-direction byte inversion). Caller\n/// has already filtered out all-null and mixed-null cases.\n///\n/// `Statistics::ByteArray` values are routed through the\n/// `Encode for str` impl after a UTF-8 check — every realistic sort\n/// prefix column (`metric_name`, `service`, tag names) is UTF-8\n/// text, and `sorted_series` itself only encodes strings, so the\n/// \"byte prefix of sorted_series\" property only holds for UTF-8\n/// values. Non-UTF-8 byte-array prefix cols would never match a\n/// `sorted_series` key in practice (sorted_series would not encode\n/// them either) and so are rejected up front.","sourceCodeStart":277,"sourceCodeEnd":313,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-parquet-engine/src/merge/streaming/region_grouping.rs#L277-L313","documentation":"After encoding all prefix column values, the code appends a u8 sentinel (the prefix length) via storekey::encode to make the per-RG key a byte-for-byte prefix of each row's sorted_series key. This error wraps any failure of that storekey encoding. Since encoding a plain u8 cannot realistically fail, this almost always signals an internal bug or a corrupted/unsupported storekey version.","triggerScenarios":"extract_rg_composite_prefix_key reaching the sentinel encode step and storekey::encode returning an Err for the u8 prefix-length value — effectively only on a storekey internal failure.","commonSituations":"A storekey version/impl change in this workspace; an unexpected buffer limitation; genuinely a can't-happen path indicating an invariant break.","solutions":["Inspect the storekey error payload in the message; if it indicates capacity or version issues, update/fix the storekey crate usage.","Audit recent changes to storekey::encode for u8/i-ordinal types.","If reproducible, add a unit test asserting storekey::encode of a u8 succeeds to catch regressions early."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if let Err(e) = extract_rg_composite_prefix_key(...) {\n    if e.to_string().contains(\"storekey encode prefix-length sentinel\") {\n        // this is an internal invariant break: capture context and report as a bug\n        log::error!(\"storekey sentinel encode failed — report with full error: {e:#}\");\n    }\n    return Err(e);\n}\n","preventionTips":["Add a unit test asserting storekey::encode of a u8 sentinel succeeds whenever the storekey crate changes.","Treat any occurrence of this error as a bug report — do not add retry/fallback logic.","Keep storekey usage minimal and pin its version in the workspace."],"tags":["storekey","encoding","internal-invariant","merge"],"backgroundTag":"internal-invariant-violation","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}