What causes "input {} is missing column '{}'" in quickwit-oss/quickwit?

Inspect the schema of the input file at the reported index and add/restore the missing column.

input {} is missing column '{}'

Error message

input {} is missing column '{}'

What it means

get_column extracts a named column from an arrow RecordBatch and converts a missing-column error from Schema::index_of into this anyhow error, reporting which input batch (by index) lacked the column. Merge order computation requires every input batch to carry the timestamp and sort columns.

Source

Thrown at quickwit/quickwit-parquet-engine/src/merge/merge_order.rs:417

            }

            transitions_remaining -= 1;
        }
    }

    Ok(boundaries)
}

/// Get a column by name from a RecordBatch, with a clear error message.
fn get_column(
    batch: &RecordBatch,
    name: &str,
    input_index: usize,
) -> Result<arrow::array::ArrayRef> {
    let idx = batch
        .schema()
        .index_of(name)
        .map_err(|_| anyhow::anyhow!("input {} is missing column '{}'", input_index, name))?;
    Ok(Arc::clone(batch.column(idx)))
}

View on GitHub (pinned to a39730c5cd)

Solutions

Inspect the schema of the input file at the reported index and add/restore the missing column.
Ensure all files in the merge set share the same arrow schema — filter out or re-write divergent files.
If using column projection, project all sort/timestamp columns in the read.

Example fix

// before: reading only a subset of columns
let batch = reader.read_columns(&["service"])?;
// after: include required merge columns
let batch = reader.read_columns(&["service", "_timestamp"])?;

Defensive patterns

Strategy: validation

Validate before calling

// before merging, verify schema parity across inputs
let schema0 = first_batch.schema();
for (i, batch) in batches.iter().enumerate() {
    for name in required_columns {
        assert!(batch.schema().index_of(name).is_ok(), "input {} missing {}", i, name);
    }
}

Type guard

fn has_column(batch: &RecordBatch, name: &str) -> bool {
    batch.schema().index_of(name).is_ok()
}

Try / catch

match compute_merge_order(&batches) {
    Err(e) if e.to_string().contains("is missing column") => {
        // re-read inputs with full projection or drop the offending file
    }
    other => other?,
}

Prevention

Project all sort/timestamp columns when reading parquet for merge.
Enforce a single arrow schema across all files in a merge set.
Detect schema drift between files before starting the merge.

When it happens

Trigger: compute_merge_order processing a RecordBatch whose schema lacks a column required by the merge order (e.g. the timestamp column or a sort key), typically the batch at the reported input index.

Common situations: Mixing parquet files with different schemas (one missing the timestamp field); schema evolution where a field was renamed; partial reads returning batches projected to fewer columns.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/613941013cd5aa1a. Report an issue: GitHub.