{"record":{"id":"4715cc99c1aaeaac","repo":"quickwit-oss/quickwit","slug":"sort-schema-mismatch-in-expected-found","errorCode":null,"errorMessage":"sort schema mismatch in {}: expected '{}', found '{}'","messagePattern":"sort schema mismatch in (.+?): expected '(.+?)', found '(.+?)'","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-parquet-engine/src/merge/mod.rs","lineNumber":377,"sourceCode":"            anyhow::anyhow!(\n                \"input file {} is missing {} metadata\",\n                path.display(),\n                PARQUET_META_SORT_FIELDS\n            )\n        })?;\n\n        match &consensus_sort_fields {\n            Some(expected) => {\n                let expected_schema = parse_sort_fields(expected)?;\n                let file_schema = parse_sort_fields(&file_sort_fields).with_context(|| {\n                    format!(\n                        \"parsing sort schema from {}: '{}'\",\n                        path.display(),\n                        file_sort_fields\n                    )\n                })?;\n                if !equivalent_schemas_for_compaction(&expected_schema, &file_schema) {\n                    bail!(\n                        \"sort schema mismatch in {}: expected '{}', found '{}'\",\n                        path.display(),\n                        expected,\n                        file_sort_fields\n                    );\n                }\n            }\n            None => {\n                // Validate the schema is parseable.\n                parse_sort_fields(&file_sort_fields).with_context(|| {\n                    format!(\n                        \"parsing sort schema from {}: '{}'\",\n                        path.display(),\n                        file_sort_fields\n                    )\n                })?;\n                consensus_sort_fields = Some(file_sort_fields.clone());\n            }","sourceCodeStart":359,"sourceCodeEnd":395,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-parquet-engine/src/merge/mod.rs#L359-L395","documentation":"extract_and_validate_input_metadata compares each input file's `qh` sort-fields KV metadata against the consensus (first file's) schema using equivalent_schemas_for_compaction. This bail fires when a later file's sort schema is not compaction-equivalent to the first's, so the files cannot be merged while preserving a single declared sort order.","triggerScenarios":"Calling merge_sorted_parquet_files with files whose PARQUET_META_SORT_FIELDS KV values parse to non-equivalent sort schemas — e.g. one file sorted by [timestamp desc] and another by [timestamp asc], or sorted on different field sets — typically after an index sort-config change.","commonSituations":"Sort configuration updated on a live index so old and new splits coexist; a partially rolled-out schema change across indexer nodes; files from different indexes accidentally collected into one compaction batch.","solutions":["Group merge candidates by compaction-equivalent sort schema (use equivalent_schemas_for_compaction as the grouping predicate) before invoking the merge.","Rebuild/re-sort legacy splits written under the old sort configuration instead of compacting them with new-schema files.","Check the index sort-config change history to determine which files are from the old era and migrate them.","Verify the file listing filters by index_uid so files from other indexes don't enter the batch."],"exampleFix":"// before\nlet batches: Vec<Vec<PathBuf>> = vec![all_paths];\nfor group in batches { merge_sorted_parquet_files(group, out, &config)?; }\n// after\nlet mut groups: HashMap<String, Vec<PathBuf>> = HashMap::new();\nfor path in all_paths {\n    let schema_key = read_sort_fields_kv(path)?; // normalize to equivalence class\n    groups.entry(schema_key).or_default().push(path);\n}\nfor group in groups.values() { merge_sorted_parquet_files(group, out, &config)?; }","handlingStrategy":"validation","validationCode":"fn same_sort_schema(paths: &[PathBuf]) -> bool {\n    let mut expected: Option<String> = None;\n    for p in paths {\n        let sf = read_sort_fields_kv(p); // read qh sort-fields KV\n        match &expected {\n            Some(e) => if !equivalent_schemas_for_compaction(&parse(e), &parse(&sf)) { return false },\n            None => expected = Some(sf),\n        }\n    }\n    true\n}","typeGuard":"fn sort_equivalence_class(path: &Path) -> Option<String> {\n    read_sort_fields_kv(path).map(|sf| normalize_sort_fields(&sf))\n}","tryCatchPattern":"match merge_sorted_parquet_files(&paths, out_dir, &config) {\n    Err(e) if e.to_string().contains(\"sort schema mismatch\") => {\n        warn!(\"sort schema drift: {e:#}; splitting batch by equivalence class\");\n        for group in group_by_sort_schema(&paths) {\n            merge_sorted_parquet_files(&group, out_dir, &config)?;\n        }\n        Ok(Vec::new())\n    }\n    other => other,\n}","preventionTips":["Group candidates by compaction-equivalent sort schema before merging.","Version the sort config and rebuild old-era splits when it changes.","Filter listings by index_uid so files from other indexes never enter a batch.","Roll out sort-config changes with a migration, not in-place."],"tags":["rust","merge","parquet","sort-schema","mismatch"],"backgroundTag":"schema-validation-failed","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}