{"record":{"id":"4bbf2d15a404df34","repo":"quickwit-oss/quickwit","slug":"input-has-rg-partition-prefix-len-expected","errorCode":null,"errorMessage":"input {} has rg_partition_prefix_len {}, expected {} — splits with different prefix lengths must not appear in the same regular merge (legacy-promotion operations bypass this check)","messagePattern":"input (.+?) has rg_partition_prefix_len (.+?), expected (.+?) — splits with different prefix lengths must not appear in the same regular merge \\(legacy-promotion operations bypass this check\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-parquet-engine/src/merge/metadata_aggregation.rs","lineNumber":105,"sourceCode":"        }\n        if input.sort_fields != first.sort_fields {\n            bail!(\n                \"input {} has sort_fields '{}', expected '{}'\",\n                i,\n                input.sort_fields,\n                first.sort_fields\n            );\n        }\n        if input.window != first.window {\n            bail!(\n                \"input {} has window {:?}, expected {:?}\",\n                i,\n                input.window,\n                first.window\n            );\n        }\n        if !mixed_prefix_ok && input.rg_partition_prefix_len != first.rg_partition_prefix_len {\n            bail!(\n                \"input {} has rg_partition_prefix_len {}, expected {} — splits with different \\\n                 prefix lengths must not appear in the same regular merge (legacy-promotion \\\n                 operations bypass this check)\",\n                i,\n                input.rg_partition_prefix_len,\n                first.rg_partition_prefix_len\n            );\n        }\n    }\n\n    // Each merge adds one to the lineage depth. The policy uses this to\n    // decide when a split is \"mature\" (reached max_merge_ops).\n    let num_merge_ops = inputs\n        .iter()\n        .map(|s| s.num_merge_ops)\n        .max()\n        .expect(\"at least one input\")\n        + 1;","sourceCodeStart":87,"sourceCodeEnd":123,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-parquet-engine/src/merge/metadata_aggregation.rs#L87-L123","documentation":"In a regular merge (mixed_prefix_ok = false), all input splits must share the same `rg_partition_prefix_len`. This bail fires when input i's prefix length differs from inputs[0]'s. Different prefix lengths mean different row-group partitioning layouts; the legacy-promotion path (mixed_prefix_ok = true) intentionally bypasses this because promotion outputs take their prefix from the writer's KV stamp rather than the inputs.","triggerScenarios":"Calling merge_parquet_split_metadata with mixed_prefix_ok=false and inputs with differing rg_partition_prefix_len — e.g. a regular compaction sweep accidentally picks up legacy (prefix-len 0) splits together with prefix-partitioned splits, instead of routing them through the legacy-promotion operation.","commonSituations":"Upgrading an index that previously had no prefix partitioning: old splits have prefix_len 0 while new ones have the configured length; merge policy written before the prefix-len field existed didn't filter on it; misconfigured legacy-promotion flag passed as false for a promotion batch.","solutions":["Set mixed_prefix_ok=true only for genuine legacy-promotion operations; verify the caller passes the right flag for the operation type.","Filter legacy (prefix-len 0) splits out of regular merge candidate sets and schedule them as promotion operations instead.","Run a one-time migration/promotion pass over pre-upgrade splits so all splits in a bucket share the same prefix length.","Add the prefix length to the merge-grouping key to guarantee homogeneity before selection."],"exampleFix":"// before\nmerge_parquet_split_metadata(&inputs, &output, false)?; // inputs contain legacy prefix-len 0 splits\n// after\nlet is_promotion = inputs.iter().any(|s| s.rg_partition_prefix_len != target_prefix_len);\nmerge_parquet_split_metadata(&inputs, &output, is_promotion)?;","handlingStrategy":"validation","validationCode":"fn prefix_len_ok(inputs: &[ParquetSplitMetadata], mixed_prefix_ok: bool) -> bool {\n    mixed_prefix_ok\n        || inputs.iter().all(|s| s.rg_partition_prefix_len == inputs[0].rg_partition_prefix_len)\n}","typeGuard":"fn prefix_homogeneous(inputs: &[ParquetSplitMetadata]) -> Option<u32> {\n    let p = inputs.first()?.rg_partition_prefix_len;\n    inputs.iter().all(|s| s.rg_partition_prefix_len == p).then_some(p)\n}","tryCatchPattern":"match merge_parquet_split_metadata(&inputs, &output, mixed_prefix_ok) {\n    Err(e) if e.to_string().contains(\"rg_partition_prefix_len\") => {\n        warn!(\"mixed prefix lengths in regular merge: {e:#}; converting to legacy promotion\");\n        merge_parquet_split_metadata(&inputs, &output, true) // only if this truly is a promotion\n    }\n    other => other?,\n}","preventionTips":["Pass mixed_prefix_ok=true only for genuine legacy-promotion operations.","Keep legacy (prefix-len 0) splits out of regular merge candidate sets.","Run the one-time promotion migration after enabling prefix partitioning.","Include rg_partition_prefix_len in the merge-grouping key."],"tags":["rust","merge","partitioning","legacy-migration","invariant-violation"],"backgroundTag":"internal-invariant-violation","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}