{"record":{"id":"84c2e03ac4f41882","repo":"quickwit-oss/quickwit","slug":"input-input-idx-rg-rg-idx-col-has-non-utf","errorCode":null,"errorMessage":"input {input_idx} rg {rg_idx} col '{}' has non-UTF-8 byte-array prefix value; only UTF-8 string prefix columns are supported (matching sorted_series's `&str` encoding)","messagePattern":"input (.+?) rg (.+?) col '(.+?)' has non-UTF-8 byte-array prefix value; only UTF-8 string prefix columns are supported \\(matching sorted_series's `&str` encoding\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-parquet-engine/src/merge/streaming/region_grouping.rs","lineNumber":370,"sourceCode":"    }\n\n    fn encode_byte_array_value(\n        min_bytes: Option<&[u8]>,\n        max_bytes: Option<&[u8]>,\n        col: &PrefixColumn,\n        rg_idx: usize,\n        input_idx: usize,\n        key: &mut Vec<u8>,\n    ) -> Result<()> {\n        let value = require_eq(\n            min_bytes.map(|b| b.to_vec()),\n            max_bytes.map(|b| b.to_vec()),\n            &col.name,\n            rg_idx,\n            input_idx,\n        )?;\n        let s = std::str::from_utf8(&value).map_err(|_| {\n            anyhow!(\n                \"input {input_idx} rg {rg_idx} col '{}' has non-UTF-8 byte-array prefix value; \\\n                 only UTF-8 string prefix columns are supported (matching sorted_series's `&str` \\\n                 encoding)\",\n                col.name,\n            )\n        })?;\n        append_prefix_col_to_key(key, col.ordinal, s, col.descending)\n    }\n\n    match stats {\n        Statistics::ByteArray(v) => {\n            encode_byte_array_value(\n                v.min_bytes_opt(),\n                v.max_bytes_opt(),\n                col,\n                rg_idx,\n                input_idx,\n                key,","sourceCodeStart":352,"sourceCodeEnd":388,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-parquet-engine/src/merge/streaming/region_grouping.rs#L352-L388","documentation":"For ByteArray prefix columns, the region-grouping code requires the constant value to be valid UTF-8, because sorted_series encodes string columns as `&str` and the per-RG key must be a byte-prefix of those row keys. A binary (non-UTF-8) ByteArray prefix value would break that correspondence, so it is rejected.","triggerScenarios":"extract_regions_from_metadata encounters a row group whose prefix-aligned ByteArray column (per min==max stats) holds bytes that are not valid UTF-8 — e.g. a binary tag value or a column written with a non-string encoding.","commonSituations":"Ingesting data with raw binary tag values into a column expected to be text; a sort-schema misconfiguration listing a binary column as a prefix column; writers encoding strings with a non-UTF-8 charset.","solutions":["Remove the binary column from the sort schema's prefix columns, or store the value as a UTF-8 string column.","Sanitize/convert the tag values to valid UTF-8 at ingestion before writing parquet.","If binary prefix support is genuinely needed, extend sorted_series's encoding — but note the current byte-prefix contract only holds for strings."],"exampleFix":"// before: sort schema references a binary ByteArray column\n// after: cast/validate at write time\nlet s = std::str::from_utf8(bytes).context(\"tag value must be UTF-8\")?;\n// write as Utf8 column instead of ByteArray","handlingStrategy":"validation","validationCode":"if col.physical_type() == \"BYTE_ARRAY\" {\n    let v = std::str::from_utf8(value_bytes)?; // fails fast pre-merge\n}\n","typeGuard":"fn is_utf8_prefix_value(stats: &Statistics) -> bool {\n    stats.min_bytes_opt()\n        .and_then(|b| std::str::from_utf8(b).ok())\n        .is_some()\n}\n","tryCatchPattern":"match result {\n    Err(e) if e.to_string().contains(\"non-UTF-8 byte-array prefix value\") => {\n        // drop the binary column from prefix cols or transcode the tag values\n        return Err(e);\n    }\n    other => other,\n}\n","preventionTips":["Restrict sort-schema prefix columns to UTF-8 text columns.","Validate tag values as UTF-8 at ingestion, not at merge time.","Add a config-time check that every ByteArray prefix column is declared as a string type."],"tags":["parquet","utf8","merge","encoding"],"backgroundTag":"invalid-argument-value","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}