RG : prefix column ' ' is not present in the file's schema

Error message

RG {rg_idx}: prefix column '{col_name}' is not present in the file's schema

What it means

verify_partition_prefix checks every row group (RG) of a parquet file for the prefix columns used in partition pruning, validating chunk_min/chunk_max consistency. This error is raised when a prefix column named in the expected prefix_columns list is not found among the file's row-group column metadata (column_path match fails) for a given RG.

Solutions

Check that prefix_columns matches the file's actual schema (parquet-tools / inspect the file metadata)
Rewrite or reindex old data files whose schema predates the prefix column addition
Verify RG-level column_path spelling matches exactly (parquet paths can include nested names)
Skip verification for legacy files if backward compatibility is intended (with care — do not hide failures)

Example fix

// before: config referencing removed column
prefix_columns = ["host", "service", "region"]  // file has only host, service
// after: align config with file schema
prefix_columns = ["host", "service"]

Defensive patterns

Strategy: validation

Validate before calling

let paths: HashSet<_> = rg.columns.iter().map(|c| c.column_path.as_str()).collect();
for col in &prefix_columns { ensure!(paths.contains(col), "missing prefix column {col}"); }

Try / catch

match verify_partition_prefix(&file, &prefix_columns) {
    Err(e) if e.to_string().contains("not present in the file's schema") => { /* legacy file: rewrite or skip */ }
    Err(e) => return Err(e),
    Ok(()) => {},
}

Prevention

Keep prefix_columns in the index config in sync with the written schema
Reindex files written before a prefix-column change
Inspect file metadata (parquet-tools) when ingesting third-party parquet

When it happens

Trigger: Calling verify_partition_prefix on a parquet file whose schema lacks one of the configured prefix columns — typically after a schema change, an index config edit, or reading a file written by an older/newer writer version.

Common situations: Adding/renaming a sort or prefix column in the index config while old data files remain; hand-edited or third-party-written parquet files with an incomplete schema; RG-level schema mismatches in multi-RG files.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/f5b6d6ee7b9612d6. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-parquet-engine/src/storage/inspect.rs:268

        _ => bail!(
            "rg_partition_prefix_len = {} but {} is missing or empty — cannot verify alignment \
             without the sort schema",
            report.rg_partition_prefix_len,
            PARQUET_META_SORT_FIELDS
        ),
    };

    let prefix_columns =
        first_n_sort_field_names(&sort_fields_str, report.rg_partition_prefix_len as usize)?;

    for (rg_idx, rg) in report.row_groups.iter().enumerate() {
        for col_name in &prefix_columns {
            let col = rg
                .columns
                .iter()
                .find(|c| c.column_path == *col_name)
                .ok_or_else(|| {
                    anyhow::anyhow!(
                        "RG {rg_idx}: prefix column '{col_name}' is not present in the file's \
                         schema",
                    )
                })?;

            match (&col.chunk_min, &col.chunk_max) {
                (Some(min), Some(max)) if min == max => {
                    // OK: column has a single value across the entire RG.
                }
                (Some(min), Some(max)) => bail!(
                    "RG {rg_idx} violates rg_partition_prefix_len={} claim: column '{col_name}' \
                     has min={min:?} != max={max:?} (must be constant across the row group)",
                    report.rg_partition_prefix_len
                ),
                _ => bail!(
                    "RG {rg_idx} violates rg_partition_prefix_len={} claim: column '{col_name}' \
                     has no chunk-level statistics",
                    report.rg_partition_prefix_len

View on GitHub (pinned to a39730c5cd)