{"record":{"id":"53e7dce234b55955","repo":"quickwit-oss/quickwit","slug":"expected-largestringarray-for-service-col-page","errorCode":null,"errorMessage":"expected LargeStringArray for service col page","messagePattern":"expected LargeStringArray for service col page","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-parquet-engine/src/merge/streaming/body_assembler.rs","lineNumber":122,"sourceCode":"            if strings.is_valid(i) {\n                out.insert(strings.value(i).to_string());\n            }\n        }\n    }\n\n    match arr.data_type() {\n        DataType::Utf8 => {\n            let strings = arr\n                .as_any()\n                .downcast_ref::<arrow::array::StringArray>()\n                .ok_or_else(|| anyhow!(\"expected StringArray for service col page\"))?;\n            extend_from_strings(strings, out);\n        }\n        DataType::LargeUtf8 => {\n            let strings = arr\n                .as_any()\n                .downcast_ref::<arrow::array::LargeStringArray>()\n                .ok_or_else(|| anyhow!(\"expected LargeStringArray for service col page\"))?;\n            for i in 0..strings.len() {\n                if strings.is_valid(i) {\n                    out.insert(strings.value(i).to_string());\n                }\n            }\n        }\n        DataType::Dictionary(key_type, value_type)\n            if matches!(value_type.as_ref(), DataType::Utf8) =>\n        {\n            // Extract the dictionary's values that are referenced by\n            // valid (non-null) keys.\n            match key_type.as_ref() {\n                DataType::Int8 => {\n                    let dict = arr.as_dictionary::<Int8Type>();\n                    if let Some(strings) = dict\n                        .values()\n                        .as_any()\n                        .downcast_ref::<arrow::array::StringArray>()","sourceCodeStart":104,"sourceCodeEnd":140,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-parquet-engine/src/merge/streaming/body_assembler.rs#L104-L140","documentation":"During service-name collection, the service column page arrived as an Arrow array whose runtime type is not LargeStringArray even though its schema declares DataType::LargeUtf8. The code downcasts the generic ArrayRef to LargeStringArray and fails with this error when the concrete array type differs. This is an internal invariant violation: the declared arrow DataType and the physical array type disagree.","triggerScenarios":"Calling collect_service_names_from_page (via next) on a record-batch stream whose service column is tagged DataType::LargeUtf8 in the schema but is backed by a different array type (e.g. StringArray/Utf8, or DictionaryArray) — typically produced by a writer or an upstream projection that did not honor the declared type.","commonSituations":"Mixing parquet→arrow readers or versions where a string column materializes as Utf8 instead of LargeUtf8; hand-built RecordBatches in tests; a schema cast step skipped or removed upstream.","solutions":["Fix the upstream producer so the service column is actually a LargeStringArray when the schema says LargeUtf8 (cast with arrow::compute::cast before consuming).","If both Utf8 and LargeUtf8 are legitimate, extend the match arm to handle DataType::Utf8 (StringArray) in addition to LargeUtf8.","Verify which reader/step produced the batch and ensure no silent schema/type drift; add a debug assert of array data_type() vs schema field type at the boundary."],"exampleFix":"// before\nlet strings = arr\n    .as_any()\n    .downcast_ref::<arrow::array::LargeStringArray>()\n    .ok_or_else(|| anyhow!(\"expected LargeStringArray for service col page\"))?;\n// after\nuse arrow::array::{Array as _, StringArray, LargeStringArray};\nlet strings: Vec<&str> = match arr.data_type() {\n    DataType::LargeUtf8 => arr.as_any().downcast_ref::<LargeStringArray>().unwrap().iter().flatten().collect(),\n    DataType::Utf8 => arr.as_any().downcast_ref::<StringArray>().unwrap().iter().flatten().collect(),\n    other => return Err(anyhow!(\"expected UTF-8 service col page, got {:?}\", other)),\n};","handlingStrategy":"type-guard","validationCode":"let field_type = batch.schema().field_with_name(\"service\")?.data_type();\nif matches!(field_type, DataType::LargeUtf8) {\n    let ok = matches!(arr.data_type(), DataType::LargeUtf8) && arr.as_any().is::<LargeStringArray>();\n}\n","typeGuard":"fn as_large_strings(arr: &dyn Array) -> Option<&LargeStringArray> {\n    (arr.data_type() == &DataType::LargeUtf8)\n        .then(|| arr.as_any().downcast_ref::<LargeStringArray>())\n        .flatten()\n}\n","tryCatchPattern":"match result {\n    Ok(names) => names,\n    Err(e) if e.to_string().contains(\"expected LargeStringArray\") => {\n        // cast then retry: let arr = arrow::compute::cast(arr, &DataType::LargeUtf8)?;\n        return Err(e.into());\n    }\n    Err(e) => return Err(e.into()),\n}\n","preventionTips":["Always build batches through a helper that asserts array data_type() matches the schema field type.","Cast string columns explicitly to LargeUtf8 at the producer boundary.","Add a debug_assert on arrow types at ingestion of each page in the merge pipeline."],"tags":["arrow","type-mismatch","parquet","merge"],"backgroundTag":"type-mismatch","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}