input {input_idx} rg {rg_idx} col '{}' has non-UTF-8 byte-ar

Error message

input {input_idx} rg {rg_idx} col '{}' has non-UTF-8 byte-array prefix value; only UTF-8 string prefix columns are supported (matching sorted_series's `&str` encoding)

What it means

For ByteArray prefix columns, the region-grouping code requires the constant value to be valid UTF-8, because sorted_series encodes string columns as `&str` and the per-RG key must be a byte-prefix of those row keys. A binary (non-UTF-8) ByteArray prefix value would break that correspondence, so it is rejected.

Source

Thrown at quickwit/quickwit-parquet-engine/src/merge/streaming/region_grouping.rs:370

    }

    fn encode_byte_array_value(
        min_bytes: Option<&[u8]>,
        max_bytes: Option<&[u8]>,
        col: &PrefixColumn,
        rg_idx: usize,
        input_idx: usize,
        key: &mut Vec<u8>,
    ) -> Result<()> {
        let value = require_eq(
            min_bytes.map(|b| b.to_vec()),
            max_bytes.map(|b| b.to_vec()),
            &col.name,
            rg_idx,
            input_idx,
        )?;
        let s = std::str::from_utf8(&value).map_err(|_| {
            anyhow!(
                "input {input_idx} rg {rg_idx} col '{}' has non-UTF-8 byte-array prefix value; \
                 only UTF-8 string prefix columns are supported (matching sorted_series's `&str` \
                 encoding)",
                col.name,
            )
        })?;
        append_prefix_col_to_key(key, col.ordinal, s, col.descending)
    }

    match stats {
        Statistics::ByteArray(v) => {
            encode_byte_array_value(
                v.min_bytes_opt(),
                v.max_bytes_opt(),
                col,
                rg_idx,
                input_idx,
                key,

View on GitHub (pinned to a39730c5cd)

Solutions

Remove the binary column from the sort schema's prefix columns, or store the value as a UTF-8 string column.
Sanitize/convert the tag values to valid UTF-8 at ingestion before writing parquet.
If binary prefix support is genuinely needed, extend sorted_series's encoding — but note the current byte-prefix contract only holds for strings.

Example fix

// before: sort schema references a binary ByteArray column
// after: cast/validate at write time
let s = std::str::from_utf8(bytes).context("tag value must be UTF-8")?;
// write as Utf8 column instead of ByteArray

Defensive patterns

Strategy: validation

Validate before calling

if col.physical_type() == "BYTE_ARRAY" {
    let v = std::str::from_utf8(value_bytes)?; // fails fast pre-merge
}

Type guard

fn is_utf8_prefix_value(stats: &Statistics) -> bool {
    stats.min_bytes_opt()
        .and_then(|b| std::str::from_utf8(b).ok())
        .is_some()
}

Try / catch

match result {
    Err(e) if e.to_string().contains("non-UTF-8 byte-array prefix value") => {
        // drop the binary column from prefix cols or transcode the tag values
        return Err(e);
    }
    other => other,
}

Prevention

Restrict sort-schema prefix columns to UTF-8 text columns.
Validate tag values as UTF-8 at ingestion, not at merge time.
Add a config-time check that every ByteArray prefix column is declared as a string type.

When it happens

Trigger: extract_regions_from_metadata encounters a row group whose prefix-aligned ByteArray column (per min==max stats) holds bytes that are not valid UTF-8 — e.g. a binary tag value or a column written with a non-string encoding.

Common situations: Ingesting data with raw binary tag values into a column expected to be text; a sort-schema misconfiguration listing a binary column as a prefix column; writers encoding strings with a non-UTF-8 charset.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/84c2e03ac4f41882. Report an issue: GitHub.