{"record":{"id":"92a471edb0a994d5","repo":"quickwit-oss/quickwit","slug":"sorted-series-column-must-be-binary-typed","errorCode":null,"errorMessage":"`{SORTED_SERIES_COLUMN}` must be Binary-typed","messagePattern":"`(.+?)` must be Binary-typed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-parquet-engine/src/merge/streaming/region_grouping.rs","lineNumber":710,"sourceCode":"    if merge_order.is_empty() {\n        return Ok(Vec::new());\n    }\n    if outputs_remaining <= 1 {\n        return Ok(vec![region.clone()]);\n    }\n\n    // Per-input sorted_series array. compute_merge_order already\n    // requires this column on every input, so a missing-column case\n    // here is a bug rather than a configuration error.\n    let mut ss_arrays: Vec<Option<&BinaryArray>> = Vec::with_capacity(aligned_sort_batches.len());\n    for batch in aligned_sort_batches {\n        match batch.schema().index_of(SORTED_SERIES_COLUMN) {\n            Ok(idx) => {\n                let arr = batch\n                    .column(idx)\n                    .as_any()\n                    .downcast_ref::<BinaryArray>()\n                    .ok_or_else(|| anyhow!(\"`{SORTED_SERIES_COLUMN}` must be Binary-typed\"))?;\n                ss_arrays.push(Some(arr));\n            }\n            Err(_) => ss_arrays.push(None),\n        }\n    }\n\n    let ss_at = |run_idx: usize| -> Option<&[u8]> {\n        let run = &merge_order[run_idx];\n        ss_arrays[run.input_index].map(|a| a.value(run.start_row))\n    };\n\n    // Walk runs, splitting before a run whose preceding sorted_series\n    // transition crosses the current target. We can only split at run\n    // boundaries (a run has constant sorted_series internally), so\n    // breaking inside a run is impossible — a giant single-series run\n    // simply lands in one output regardless of size.\n    let mut splits: Vec<std::ops::Range<usize>> = Vec::new();\n    let mut current_start: usize = 0;","sourceCodeStart":692,"sourceCodeEnd":728,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-parquet-engine/src/merge/streaming/region_grouping.rs#L692-L728","documentation":"split_region_at_sorted_series expects the SORTED_SERIES_COLUMN to be a BinaryArray when present in the batch schema. If the column exists but its physical array type is not BinaryArray (e.g. LargeBinaryArray or Utf8), the downcast fails with this error. It is an internal contract check between the sorted_series writer and the streaming merge reader.","triggerScenarios":"streaming_merge_sorted_parquet_files consuming record batches that contain the sorted_series column produced/typed as something other than DataType::Binary — e.g. a schema change to LargeBinary upstream, or the column built as Utf8 instead of Binary.","commonSituations":"A producer that changed the column type (Utf8/LargeBinary) without updating the merge path; hand-built test batches with the wrong array type; a parquet round-trip that altered the physical type.","solutions":["Ensure append_sorted_series_column is the single producer of this column (it emits DataType::Binary) and that no intermediate step recasts it.","Add an arrow::compute::cast to Binary before split_region_at_sorted_series if a recast is unavoidable.","Check for a duplicate ad-hoc producer of SORTED_SERIES_COLUMN with the wrong type."],"exampleFix":"// before\nlet arr = batch.column(idx).as_any().downcast_ref::<BinaryArray>()...;\n// after\nlet col = arrow::compute::cast(batch.column(idx), &DataType::Binary)?;\nlet arr = col.as_any().downcast_ref::<BinaryArray>().ok_or_else(|| anyhow!(\"cast failed\"))?;","handlingStrategy":"type-guard","validationCode":"if let Ok(idx) = batch.schema().index_of(SORTED_SERIES_COLUMN) {\n    debug_assert_eq!(batch.schema().field(idx).data_type(), &DataType::Binary);\n}\n","typeGuard":"fn is_binary_sorted_series(batch: &RecordBatch) -> bool {\n    batch.schema().index_of(SORTED_SERIES_COLUMN)\n        .map(|i| batch.column(i).data_type() == &DataType::Binary\n            && batch.column(i).as_any().is::<BinaryArray>())\n        .unwrap_or(true)\n}\n","tryCatchPattern":"match result {\n    Err(e) if e.to_string().contains(\"must be Binary-typed\") => {\n        // recast and retry\n        let cast = arrow::compute::cast(&col, &DataType::Binary)?;\n        // retry split with cast batch\n    }\n    other => other,\n}\n","preventionTips":["Produce SORTED_SERIES_COLUMN only via append_sorted_series_column, which emits DataType::Binary.","Avoid arrow::compute::cast or projections that change the column's physical type mid-pipeline.","Assert the column type in any test fixture that hand-builds batches containing sorted_series."],"tags":["arrow","type-mismatch","merge","schema"],"backgroundTag":"type-mismatch","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}