{"record":{"id":"8e63a32b9acc793d","repo":"quickwit-oss/quickwit","slug":"dictionary-values-are-not-utf8","errorCode":null,"errorMessage":"dictionary values are not Utf8","messagePattern":"dictionary values are not Utf8","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-parquet-engine/src/sorted_series/mod.rs","lineNumber":333,"sourceCode":"/// columns in the sort schema must be string-typed.\nfn extract_string_value(array: &dyn Array, row: usize) -> Result<&str> {\n    debug_assert!(\n        !array.is_null(row),\n        \"caller must check is_null before extract_string_value\"\n    );\n\n    match array.data_type() {\n        DataType::Dictionary(_, _) => {\n            let dict = array\n                .as_any()\n                .downcast_ref::<DictionaryArray<Int32Type>>()\n                .ok_or_else(|| anyhow!(\"dictionary downcast failed for {:?}\", array.data_type()))?;\n            let key_idx = dict.keys().value(row) as usize;\n            let values = dict.values();\n            let str_values = values\n                .as_any()\n                .downcast_ref::<StringArray>()\n                .ok_or_else(|| anyhow!(\"dictionary values are not Utf8\"))?;\n            Ok(str_values.value(key_idx))\n        }\n        DataType::Utf8 => {\n            let arr = array\n                .as_any()\n                .downcast_ref::<StringArray>()\n                .ok_or_else(|| anyhow!(\"Utf8 downcast failed\"))?;\n            Ok(arr.value(row))\n        }\n        other => Err(anyhow!(\n            \"unsupported data type {:?} in sort schema tag column — only Dictionary(Int32, Utf8) \\\n             and Utf8 are supported\",\n            other\n        )),\n    }\n}\n\n/// Extract an i64 value from a column at the given row.","sourceCodeStart":315,"sourceCodeEnd":351,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-parquet-engine/src/sorted_series/mod.rs#L315-L351","documentation":"After successfully downcasting to DictionaryArray<Int32Type>, extract_string_value expects the dictionary's values array to be a StringArray (Utf8). If the dictionary was built over another value type (Binary, LargeUtf8, etc.), the downcast to StringArray fails and this error is raised.","triggerScenarios":"encode_row_key processes a Dictionary(Int32, X) tag column where X is not Utf8 — e.g. Binary or LargeUtf8 dictionary values — so values.as_any().downcast_ref::<StringArray>() returns None.","commonSituations":"Parquet files with BYTE_ARRAY dictionary columns typed as Binary instead of Utf8, or files written with LargeUtf8 logical types by other Arrow-based writers.","solutions":["Verify the parquet column's logical type is UTF8 (not BYTE_ARRAY without annotation) at write time","Cast the dictionary values to Utf8 before extraction (arrow::compute::cast)","Extend extract_string_value to also handle LargeUtf8/Binary dictionary values","Use the sort schema definition to reject unsupported tag column types at ingest rather than at key encoding"],"exampleFix":"// before\nlet str_values = values.as_any().downcast_ref::<StringArray>().ok_or_else(|| anyhow!(\"dictionary values are not Utf8\"))?;\n// after\nlet str_values = values.as_any().downcast_ref::<StringArray>()\n    .or_else(|| values.as_any().downcast_ref::<LargeStringArray>().map(large_to_utf8))\n    .ok_or_else(|| anyhow!(\"dictionary values are not Utf8 (got {:?})\", values.data_type()))?;","handlingStrategy":"type-guard","validationCode":"if let DataType::Dictionary(_, v) = array.data_type() {\n    ensure!(**v == DataType::Utf8, \"tag dict values must be Utf8, got {:?}\", v);\n}","typeGuard":"fn has_utf8_dict_values(array: &dyn Array) -> bool {\n    match array.data_type() {\n        DataType::Dictionary(_, v) => **v == DataType::Utf8,\n        _ => true,\n    }\n}","tryCatchPattern":"if let Err(e) = encode_row_key(...) {\n    if e.to_string().contains(\"not Utf8\") { /* cast values to Utf8 and retry */ }\n}","preventionTips":["Ensure parquet columns carry the UTF8 logical annotation","Cast Binary/LargeUtf8 dictionary values to Utf8 at ingestion","Validate tag column types against the sort schema on file open"],"tags":["arrow","parquet","dictionary-encoding","utf8"],"backgroundTag":"type-mismatch","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}