{"record":{"id":"32c364ab120f928b","repo":"quickwit-oss/quickwit","slug":"rg-partition-prefix-len-mismatch-in-expected","errorCode":null,"errorMessage":"rg_partition_prefix_len mismatch in {}: expected {}, found {} — splits with different prefix lengths must not appear in the same merge","messagePattern":"rg_partition_prefix_len mismatch in (.+?): expected (.+?), found (.+?) — splits with different prefix lengths must not appear in the same merge","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-parquet-engine/src/merge/mod.rs","lineNumber":465,"sourceCode":"            .with_context(|| format!(\"parsing num_merge_ops from {}\", path.display()))?\n            .unwrap_or(0);\n\n        if file_merge_ops > max_merge_ops {\n            max_merge_ops = file_merge_ops;\n        }\n\n        // Row group partition prefix length: must be consistent across all\n        // inputs. Absent KV → 0 (legacy default; no alignment claim).\n        let file_prefix_len = find_kv(PARQUET_META_RG_PARTITION_PREFIX_LEN)\n            .map(|s| s.parse::<u32>())\n            .transpose()\n            .with_context(|| format!(\"parsing rg_partition_prefix_len from {}\", path.display()))?\n            .unwrap_or(0);\n\n        match consensus_prefix_len {\n            Some(expected) => {\n                if file_prefix_len != expected {\n                    bail!(\n                        \"rg_partition_prefix_len mismatch in {}: expected {}, found {} — splits \\\n                         with different prefix lengths must not appear in the same merge\",\n                        path.display(),\n                        expected,\n                        file_prefix_len\n                    );\n                }\n            }\n            None => {\n                consensus_prefix_len = Some(file_prefix_len);\n            }\n        }\n    }\n\n    Ok(InputMetadata {\n        sort_fields: consensus_sort_fields.expect(\"at least one input required\"),\n        window_start_secs: consensus_window_start.expect(\"at least one input required\"),\n        window_duration_secs: consensus_window_duration.unwrap_or(0),","sourceCodeStart":447,"sourceCodeEnd":483,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-parquet-engine/src/merge/mod.rs#L447-L483","documentation":"All merge inputs must have been produced with the same row-group partition prefix length (`rg_partition_prefix_len`). If a file's metadata value differs from the consensus (missing defaults to 0), the merge is aborted because row groups with different prefix lengths are not guaranteed to be aligned and cannot be merged positionally.","triggerScenarios":"Calling merge_sorted_parquet_files_impl with files whose `rg_partition_prefix_len` KV metadata values disagree, e.g. mixing legacy files (key absent, 0) with PR-5-style aligned files.","commonSituations":"Upgrading Quickwit and merging pre-upgrade splits with post-upgrade ones; a compaction scheduler that does not filter splits by prefix length; manually crafted merge tasks including heterogeneous splits.","solutions":["Ensure the merge operation only contains splits sharing the same rg_partition_prefix_len value.","Route legacy multi-row-group files (prefix_len 0) through the legacy/PR-5 adapter path instead of the streaming merger.","Re-write old splits through the current writer so they are stamped with the current prefix length.","Fix the control plane / merge scheduler to include prefix_len in its split-grouping key."],"exampleFix":"// before: mixing legacy and aligned splits\nexecute_merge_operation(&op, load(op.splits), &out, &cfg).await?;\n// after: assert homogeneity before scheduling\nlet lens: HashSet<_> = op.splits.iter().map(|s| s.rg_partition_prefix_len).collect();\nanyhow::ensure!(lens.len() == 1, \"mixed prefix lengths in merge task\");\nexecute_merge_operation(&op, load(op.splits), &out, &cfg).await?;","handlingStrategy":"validation","validationCode":"fn prefix_lens_consistent(splits: &[Split]) -> anyhow::Result<()> {\n    let lens: HashSet<u32> = splits.iter().map(|s| s.rg_partition_prefix_len).collect();\n    anyhow::ensure!(lens.len() == 1, \"splits with different rg_partition_prefix_len grouped together\");\n    Ok(())\n}","typeGuard":null,"tryCatchPattern":"match execute_merge_operation(...).await {\n    Err(e) if e.to_string().contains(\"rg_partition_prefix_len mismatch\") => {\n        // split the merge task into per-prefix-len groups and retry each\n    }\n    other => other?,\n}","preventionTips":["Partition merge tasks by prefix length as well as index and window.","After upgrades, run a rewrite pass to stamp legacy files with current metadata.","Check file metadata when importing external splits."],"tags":["merge","parquet","row-group","alignment"],"backgroundTag":"schema-validation-failed","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}