quickwit-oss/quickwit · error

dictionary values are not Utf8

Error message

dictionary values are not Utf8

What it means

After successfully downcasting to DictionaryArray<Int32Type>, extract_string_value expects the dictionary's values array to be a StringArray (Utf8). If the dictionary was built over another value type (Binary, LargeUtf8, etc.), the downcast to StringArray fails and this error is raised.

Solutions

1. Verify the parquet column's logical type is UTF8 (not BYTE_ARRAY without annotation) at write time
2. Cast the dictionary values to Utf8 before extraction (arrow::compute::cast)
3. Extend extract_string_value to also handle LargeUtf8/Binary dictionary values
4. Use the sort schema definition to reject unsupported tag column types at ingest rather than at key encoding

Example fix

// before
let str_values = values.as_any().downcast_ref::<StringArray>().ok_or_else(|| anyhow!("dictionary values are not Utf8"))?;
// after
let str_values = values.as_any().downcast_ref::<StringArray>()
    .or_else(|| values.as_any().downcast_ref::<LargeStringArray>().map(large_to_utf8))
    .ok_or_else(|| anyhow!("dictionary values are not Utf8 (got {:?})", values.data_type()))?;

if let DataType::Dictionary(_, v) = array.data_type() {
    ensure!(**v == DataType::Utf8, "tag dict values must be Utf8, got {:?}", v);
}

fn has_utf8_dict_values(array: &dyn Array) -> bool {
    match array.data_type() {
        DataType::Dictionary(_, v) => **v == DataType::Utf8,
        _ => true,
    }
}

if let Err(e) = encode_row_key(...) {
    if e.to_string().contains("not Utf8") { /* cast values to Utf8 and retry */ }
}

When it happens

Trigger: encode_row_key processes a Dictionary(Int32, X) tag column where X is not Utf8 — e.g. Binary or LargeUtf8 dictionary values — so values.as_any().downcast_ref::<StringArray>() returns None.

Common situations: Parquet files with BYTE_ARRAY dictionary columns typed as Binary instead of Utf8, or files written with LargeUtf8 logical types by other Arrow-based writers.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• dictionary downcast failed for
• expected LargeStringArray for service col page
• expected StringArray for service col page
• input rg col ' ' has non-UTF-8 byte-array prefix value…
• input is missing column

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/8e63a32b9acc793d. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-parquet-engine/src/sorted_series/mod.rs:333

/// columns in the sort schema must be string-typed.
fn extract_string_value(array: &dyn Array, row: usize) -> Result<&str> {
    debug_assert!(
        !array.is_null(row),
        "caller must check is_null before extract_string_value"
    );

    match array.data_type() {
        DataType::Dictionary(_, _) => {
            let dict = array
                .as_any()
                .downcast_ref::<DictionaryArray<Int32Type>>()
                .ok_or_else(|| anyhow!("dictionary downcast failed for {:?}", array.data_type()))?;
            let key_idx = dict.keys().value(row) as usize;
            let values = dict.values();
            let str_values = values
                .as_any()
                .downcast_ref::<StringArray>()
                .ok_or_else(|| anyhow!("dictionary values are not Utf8"))?;
            Ok(str_values.value(key_idx))
        }
        DataType::Utf8 => {
            let arr = array
                .as_any()
                .downcast_ref::<StringArray>()
                .ok_or_else(|| anyhow!("Utf8 downcast failed"))?;
            Ok(arr.value(row))
        }
        other => Err(anyhow!(
            "unsupported data type {:?} in sort schema tag column — only Dictionary(Int32, Utf8) \
             and Utf8 are supported",
            other
        )),
    }
}

/// Extract an i64 value from a column at the given row.

View on GitHub (pinned to a39730c5cd)