quickwit-oss/quickwit · error

input is missing column

Error message

input {} is missing column '{}'

What it means

get_column extracts a named column from an arrow RecordBatch and converts a missing-column error from Schema::index_of into this anyhow error, reporting which input batch (by index) lacked the column. Merge order computation requires every input batch to carry the timestamp and sort columns.

Solutions

1. Inspect the schema of the input file at the reported index and add/restore the missing column.
2. Ensure all files in the merge set share the same arrow schema — filter out or re-write divergent files.
3. If using column projection, project all sort/timestamp columns in the read.

Example fix

// before: reading only a subset of columns
let batch = reader.read_columns(&["service"])?;
// after: include required merge columns
let batch = reader.read_columns(&["service", "_timestamp"])?;

// before merging, verify schema parity across inputs
let schema0 = first_batch.schema();
for (i, batch) in batches.iter().enumerate() {
    for name in required_columns {
        assert!(batch.schema().index_of(name).is_ok(), "input {} missing {}", i, name);
    }
}

fn has_column(batch: &RecordBatch, name: &str) -> bool {
    batch.schema().index_of(name).is_ok()
}

match compute_merge_order(&batches) {
    Err(e) if e.to_string().contains("is missing column") => {
        // re-read inputs with full projection or drop the offending file
    }
    other => other?,
}

When it happens

Trigger: compute_merge_order processing a RecordBatch whose schema lacks a column required by the merge order (e.g. the timestamp column or a sort key), typically the batch at the reported input index.

Common situations: Mixing parquet files with different schemas (one missing the timestamp field); schema evolution where a field was renamed; partial reads returning batches projected to fewer columns.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• sort schema ' ' does not contain a timestamp column
• expected LargeStringArray for service col page
• input file is missing the ' ' column
• ` ` must be Binary-typed
• append column

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/613941013cd5aa1a. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-parquet-engine/src/merge/merge_order.rs:417

            }

            transitions_remaining -= 1;
        }
    }

    Ok(boundaries)
}

/// Get a column by name from a RecordBatch, with a clear error message.
fn get_column(
    batch: &RecordBatch,
    name: &str,
    input_index: usize,
) -> Result<arrow::array::ArrayRef> {
    let idx = batch
        .schema()
        .index_of(name)
        .map_err(|_| anyhow::anyhow!("input {} is missing column '{}'", input_index, name))?;
    Ok(Arc::clone(batch.column(idx)))
}

View on GitHub (pinned to a39730c5cd)