quickwit-oss/quickwit · error

`{SORTED_SERIES_COLUMN}` must be Binary-typed

Error message

`{SORTED_SERIES_COLUMN}` must be Binary-typed

What it means

split_region_at_sorted_series expects the SORTED_SERIES_COLUMN to be a BinaryArray when present in the batch schema. If the column exists but its physical array type is not BinaryArray (e.g. LargeBinaryArray or Utf8), the downcast fails with this error. It is an internal contract check between the sorted_series writer and the streaming merge reader.

Source

Thrown at quickwit/quickwit-parquet-engine/src/merge/streaming/region_grouping.rs:710

    if merge_order.is_empty() {
        return Ok(Vec::new());
    }
    if outputs_remaining <= 1 {
        return Ok(vec![region.clone()]);
    }

    // Per-input sorted_series array. compute_merge_order already
    // requires this column on every input, so a missing-column case
    // here is a bug rather than a configuration error.
    let mut ss_arrays: Vec<Option<&BinaryArray>> = Vec::with_capacity(aligned_sort_batches.len());
    for batch in aligned_sort_batches {
        match batch.schema().index_of(SORTED_SERIES_COLUMN) {
            Ok(idx) => {
                let arr = batch
                    .column(idx)
                    .as_any()
                    .downcast_ref::<BinaryArray>()
                    .ok_or_else(|| anyhow!("`{SORTED_SERIES_COLUMN}` must be Binary-typed"))?;
                ss_arrays.push(Some(arr));
            }
            Err(_) => ss_arrays.push(None),
        }
    }

    let ss_at = |run_idx: usize| -> Option<&[u8]> {
        let run = &merge_order[run_idx];
        ss_arrays[run.input_index].map(|a| a.value(run.start_row))
    };

    // Walk runs, splitting before a run whose preceding sorted_series
    // transition crosses the current target. We can only split at run
    // boundaries (a run has constant sorted_series internally), so
    // breaking inside a run is impossible — a giant single-series run
    // simply lands in one output regardless of size.
    let mut splits: Vec<std::ops::Range<usize>> = Vec::new();
    let mut current_start: usize = 0;

View on GitHub (pinned to a39730c5cd)

Solutions

  1. Ensure append_sorted_series_column is the single producer of this column (it emits DataType::Binary) and that no intermediate step recasts it.
  2. Add an arrow::compute::cast to Binary before split_region_at_sorted_series if a recast is unavoidable.
  3. Check for a duplicate ad-hoc producer of SORTED_SERIES_COLUMN with the wrong type.

Example fix

// before
let arr = batch.column(idx).as_any().downcast_ref::<BinaryArray>()...;
// after
let col = arrow::compute::cast(batch.column(idx), &DataType::Binary)?;
let arr = col.as_any().downcast_ref::<BinaryArray>().ok_or_else(|| anyhow!("cast failed"))?;
Defensive patterns

Strategy: type-guard

Validate before calling

if let Ok(idx) = batch.schema().index_of(SORTED_SERIES_COLUMN) {
    debug_assert_eq!(batch.schema().field(idx).data_type(), &DataType::Binary);
}

Type guard

fn is_binary_sorted_series(batch: &RecordBatch) -> bool {
    batch.schema().index_of(SORTED_SERIES_COLUMN)
        .map(|i| batch.column(i).data_type() == &DataType::Binary
            && batch.column(i).as_any().is::<BinaryArray>())
        .unwrap_or(true)
}

Try / catch

match result {
    Err(e) if e.to_string().contains("must be Binary-typed") => {
        // recast and retry
        let cast = arrow::compute::cast(&col, &DataType::Binary)?;
        // retry split with cast batch
    }
    other => other,
}

Prevention

When it happens

Trigger: streaming_merge_sorted_parquet_files consuming record batches that contain the sorted_series column produced/typed as something other than DataType::Binary — e.g. a schema change to LargeBinary upstream, or the column built as Utf8 instead of Binary.

Common situations: A producer that changed the column type (Utf8/LargeBinary) without updating the merge path; hand-built test batches with the wrong array type; a parquet round-trip that altered the physical type.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/92a471edb0a994d5. Report an issue: GitHub.