quickwit-oss/quickwit · error

sort schema mismatch in {}: expected '{}', found '{}'

Error message

sort schema mismatch in {}: expected '{}', found '{}'

What it means

extract_and_validate_input_metadata compares each input file's `qh` sort-fields KV metadata against the consensus (first file's) schema using equivalent_schemas_for_compaction. This bail fires when a later file's sort schema is not compaction-equivalent to the first's, so the files cannot be merged while preserving a single declared sort order.

Source

Thrown at quickwit/quickwit-parquet-engine/src/merge/mod.rs:377

            anyhow::anyhow!(
                "input file {} is missing {} metadata",
                path.display(),
                PARQUET_META_SORT_FIELDS
            )
        })?;

        match &consensus_sort_fields {
            Some(expected) => {
                let expected_schema = parse_sort_fields(expected)?;
                let file_schema = parse_sort_fields(&file_sort_fields).with_context(|| {
                    format!(
                        "parsing sort schema from {}: '{}'",
                        path.display(),
                        file_sort_fields
                    )
                })?;
                if !equivalent_schemas_for_compaction(&expected_schema, &file_schema) {
                    bail!(
                        "sort schema mismatch in {}: expected '{}', found '{}'",
                        path.display(),
                        expected,
                        file_sort_fields
                    );
                }
            }
            None => {
                // Validate the schema is parseable.
                parse_sort_fields(&file_sort_fields).with_context(|| {
                    format!(
                        "parsing sort schema from {}: '{}'",
                        path.display(),
                        file_sort_fields
                    )
                })?;
                consensus_sort_fields = Some(file_sort_fields.clone());
            }

View on GitHub (pinned to a39730c5cd)

Solutions

  1. Group merge candidates by compaction-equivalent sort schema (use equivalent_schemas_for_compaction as the grouping predicate) before invoking the merge.
  2. Rebuild/re-sort legacy splits written under the old sort configuration instead of compacting them with new-schema files.
  3. Check the index sort-config change history to determine which files are from the old era and migrate them.
  4. Verify the file listing filters by index_uid so files from other indexes don't enter the batch.

Example fix

// before
let batches: Vec<Vec<PathBuf>> = vec![all_paths];
for group in batches { merge_sorted_parquet_files(group, out, &config)?; }
// after
let mut groups: HashMap<String, Vec<PathBuf>> = HashMap::new();
for path in all_paths {
    let schema_key = read_sort_fields_kv(path)?; // normalize to equivalence class
    groups.entry(schema_key).or_default().push(path);
}
for group in groups.values() { merge_sorted_parquet_files(group, out, &config)?; }
Defensive patterns

Strategy: validation

Validate before calling

fn same_sort_schema(paths: &[PathBuf]) -> bool {
    let mut expected: Option<String> = None;
    for p in paths {
        let sf = read_sort_fields_kv(p); // read qh sort-fields KV
        match &expected {
            Some(e) => if !equivalent_schemas_for_compaction(&parse(e), &parse(&sf)) { return false },
            None => expected = Some(sf),
        }
    }
    true
}

Type guard

fn sort_equivalence_class(path: &Path) -> Option<String> {
    read_sort_fields_kv(path).map(|sf| normalize_sort_fields(&sf))
}

Try / catch

match merge_sorted_parquet_files(&paths, out_dir, &config) {
    Err(e) if e.to_string().contains("sort schema mismatch") => {
        warn!("sort schema drift: {e:#}; splitting batch by equivalence class");
        for group in group_by_sort_schema(&paths) {
            merge_sorted_parquet_files(&group, out_dir, &config)?;
        }
        Ok(Vec::new())
    }
    other => other,
}

Prevention

When it happens

Trigger: Calling merge_sorted_parquet_files with files whose PARQUET_META_SORT_FIELDS KV values parse to non-equivalent sort schemas — e.g. one file sorted by [timestamp desc] and another by [timestamp asc], or sorted on different field sets — typically after an index sort-config change.

Common situations: Sort configuration updated on a live index so old and new splits coexist; a partially rolled-out schema change across indexer nodes; files from different indexes accidentally collected into one compaction batch.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/4715cc99c1aaeaac. Report an issue: GitHub.