type conflict for column

Error message

type conflict for column '{}': input 0 has {:?}, input {} has {:?} (normalized: {:?} vs {:?})

What it means

While building the union schema, each column's normalized data type is tracked across all input batches. If two inputs contain the same column name with different (post-normalization) types, the batches cannot be combined into one union column, so the alignment bails naming the conflicting column, the input indices, and both types.

Solutions

Find which splits contain the divergent column type and re-index them so the field has a consistent type.
Update the index config / doc mapper so the field type matches the data actually ingested.
Extend normalize_type if the conflicting types should be considered compatible (e.g. widening numerics).
Reject or quarantine documents producing the off-type values at ingestion time.

Example fix

// before: field 'status' changed Int64 -> Utf8 between writer versions; merge fails
// after: pin the field type in the doc mapper
// quickwit.yaml index config
// field_mappings:
//   - name: status
//     type: u64   # keep stable across writer versions

Defensive patterns

Strategy: validation

Validate before calling

fn schemas_compatible(schemas: &[SchemaRef]) -> anyhow::Result<()> {
    let mut seen: HashMap<String, DataType> = HashMap::new();
    for s in schemas {
        for f in s.fields() {
            let dt = normalize_type(f.data_type());
            match seen.insert(f.name().clone(), dt.clone()) {
                Some(prev) if prev != dt => anyhow::bail!("column '{}' type conflict: {:?} vs {:?}", f.name(), prev, dt),
                _ => {}
            }
        }
    }
    Ok(())
}

Try / catch

match align_inputs_to_union_schema(&batches, &sort) {
    Err(e) if e.to_string().contains("type conflict for column") => {
        // identify the offending splits, re-index them, then retry
    }
    other => other.map(|_| ()),
}

Prevention

Pin field types in the doc mapper and never change them without a migration.
Validate document field types at ingestion time.
Before merging across writer versions, compare split schemas.

When it happens

Trigger: Calling align_inputs_to_union_schema where input batches disagree on a column's arrow type (e.g. Int64 vs Utf8 for the same field name) after normalize_type.

Common situations: Splits written with different schema versions (a field type changed between releases); documents with inconsistent types ingested into the same index; JSON ingestion coercing values differently per batch.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/94a20430adb69d8a. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-parquet-engine/src/merge/schema.rs:85

    // The previous version always defaulted new fields to nullable on
    // first sight, which broke columns whose nullability must be
    // preserved (e.g. `List<Float64>` — the writer's non-nullable-
    // list contract requires the union field to stay non-nullable).
    struct FieldInfo {
        normalized_type: DataType,
        any_nullable: bool,
        appears_in: usize,
    }
    let mut field_map: BTreeMap<String, FieldInfo> = BTreeMap::new();

    for (input_idx, batch) in inputs.iter().enumerate() {
        for field in batch.schema().fields() {
            let normalized_type = normalize_type(field.data_type());

            match field_map.get_mut(field.name().as_str()) {
                Some(existing) => {
                    if existing.normalized_type != normalized_type {
                        bail!(
                            "type conflict for column '{}': input 0 has {:?}, input {} has {:?} \
                             (normalized: {:?} vs {:?})",
                            field.name(),
                            existing.normalized_type,
                            input_idx,
                            field.data_type(),
                            existing.normalized_type,
                            normalized_type,
                        );
                    }
                    if field.is_nullable() {
                        existing.any_nullable = true;
                    }
                    existing.appears_in += 1;
                }
                None => {
                    field_map.insert(
                        field.name().clone(),

View on GitHub (pinned to a39730c5cd)