quickwit-oss/quickwit · error

expected LargeStringArray for service col page

Error message

expected LargeStringArray for service col page

What it means

During service-name collection, the service column page arrived as an Arrow array whose runtime type is not LargeStringArray even though its schema declares DataType::LargeUtf8. The code downcasts the generic ArrayRef to LargeStringArray and fails with this error when the concrete array type differs. This is an internal invariant violation: the declared arrow DataType and the physical array type disagree.

Source

Thrown at quickwit/quickwit-parquet-engine/src/merge/streaming/body_assembler.rs:122

            if strings.is_valid(i) {
                out.insert(strings.value(i).to_string());
            }
        }
    }

    match arr.data_type() {
        DataType::Utf8 => {
            let strings = arr
                .as_any()
                .downcast_ref::<arrow::array::StringArray>()
                .ok_or_else(|| anyhow!("expected StringArray for service col page"))?;
            extend_from_strings(strings, out);
        }
        DataType::LargeUtf8 => {
            let strings = arr
                .as_any()
                .downcast_ref::<arrow::array::LargeStringArray>()
                .ok_or_else(|| anyhow!("expected LargeStringArray for service col page"))?;
            for i in 0..strings.len() {
                if strings.is_valid(i) {
                    out.insert(strings.value(i).to_string());
                }
            }
        }
        DataType::Dictionary(key_type, value_type)
            if matches!(value_type.as_ref(), DataType::Utf8) =>
        {
            // Extract the dictionary's values that are referenced by
            // valid (non-null) keys.
            match key_type.as_ref() {
                DataType::Int8 => {
                    let dict = arr.as_dictionary::<Int8Type>();
                    if let Some(strings) = dict
                        .values()
                        .as_any()
                        .downcast_ref::<arrow::array::StringArray>()

View on GitHub (pinned to a39730c5cd)

Solutions

1. Fix the upstream producer so the service column is actually a LargeStringArray when the schema says LargeUtf8 (cast with arrow::compute::cast before consuming).
2. If both Utf8 and LargeUtf8 are legitimate, extend the match arm to handle DataType::Utf8 (StringArray) in addition to LargeUtf8.
3. Verify which reader/step produced the batch and ensure no silent schema/type drift; add a debug assert of array data_type() vs schema field type at the boundary.

Example fix

// before
let strings = arr
    .as_any()
    .downcast_ref::<arrow::array::LargeStringArray>()
    .ok_or_else(|| anyhow!("expected LargeStringArray for service col page"))?;
// after
use arrow::array::{Array as _, StringArray, LargeStringArray};
let strings: Vec<&str> = match arr.data_type() {
    DataType::LargeUtf8 => arr.as_any().downcast_ref::<LargeStringArray>().unwrap().iter().flatten().collect(),
    DataType::Utf8 => arr.as_any().downcast_ref::<StringArray>().unwrap().iter().flatten().collect(),
    other => return Err(anyhow!("expected UTF-8 service col page, got {:?}", other)),
};

let field_type = batch.schema().field_with_name("service")?.data_type();
if matches!(field_type, DataType::LargeUtf8) {
    let ok = matches!(arr.data_type(), DataType::LargeUtf8) && arr.as_any().is::<LargeStringArray>();
}

fn as_large_strings(arr: &dyn Array) -> Option<&LargeStringArray> {
    (arr.data_type() == &DataType::LargeUtf8)
        .then(|| arr.as_any().downcast_ref::<LargeStringArray>())
        .flatten()
}

match result {
    Ok(names) => names,
    Err(e) if e.to_string().contains("expected LargeStringArray") => {
        // cast then retry: let arr = arrow::compute::cast(arr, &DataType::LargeUtf8)?;
        return Err(e.into());
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Calling collect_service_names_from_page (via next) on a record-batch stream whose service column is tagged DataType::LargeUtf8 in the schema but is backed by a different array type (e.g. StringArray/Utf8, or DictionaryArray) — typically produced by a writer or an upstream projection that did not honor the declared type.

Common situations: Mixing parquet→arrow readers or versions where a string column materializes as Utf8 instead of LargeUtf8; hand-built RecordBatches in tests; a schema cast step skipped or removed upstream.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• sort schema '{}' does not contain a timestamp column
• input {} is missing column '{}'
• expected StringArray for service col page
• `{SORTED_SERIES_COLUMN}` must be Binary-typed
• timestamp_secs must be UInt64 or Int64 for MC-3 check

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/53e7dce234b55955. Report an issue: GitHub.