quickwit-oss/quickwit · error

input {} has rg_partition_prefix_len {}, expected {} — split

Error message

input {} has rg_partition_prefix_len {}, expected {} — splits with different prefix lengths must not appear in the same regular merge (legacy-promotion operations bypass this check)

What it means

In a regular merge (mixed_prefix_ok = false), all input splits must share the same `rg_partition_prefix_len`. This bail fires when input i's prefix length differs from inputs[0]'s. Different prefix lengths mean different row-group partitioning layouts; the legacy-promotion path (mixed_prefix_ok = true) intentionally bypasses this because promotion outputs take their prefix from the writer's KV stamp rather than the inputs.

Source

Thrown at quickwit/quickwit-parquet-engine/src/merge/metadata_aggregation.rs:105

        }
        if input.sort_fields != first.sort_fields {
            bail!(
                "input {} has sort_fields '{}', expected '{}'",
                i,
                input.sort_fields,
                first.sort_fields
            );
        }
        if input.window != first.window {
            bail!(
                "input {} has window {:?}, expected {:?}",
                i,
                input.window,
                first.window
            );
        }
        if !mixed_prefix_ok && input.rg_partition_prefix_len != first.rg_partition_prefix_len {
            bail!(
                "input {} has rg_partition_prefix_len {}, expected {} — splits with different \
                 prefix lengths must not appear in the same regular merge (legacy-promotion \
                 operations bypass this check)",
                i,
                input.rg_partition_prefix_len,
                first.rg_partition_prefix_len
            );
        }
    }

    // Each merge adds one to the lineage depth. The policy uses this to
    // decide when a split is "mature" (reached max_merge_ops).
    let num_merge_ops = inputs
        .iter()
        .map(|s| s.num_merge_ops)
        .max()
        .expect("at least one input")
        + 1;

View on GitHub (pinned to a39730c5cd)

Solutions

1. Set mixed_prefix_ok=true only for genuine legacy-promotion operations; verify the caller passes the right flag for the operation type.
2. Filter legacy (prefix-len 0) splits out of regular merge candidate sets and schedule them as promotion operations instead.
3. Run a one-time migration/promotion pass over pre-upgrade splits so all splits in a bucket share the same prefix length.
4. Add the prefix length to the merge-grouping key to guarantee homogeneity before selection.

Example fix

// before
merge_parquet_split_metadata(&inputs, &output, false)?; // inputs contain legacy prefix-len 0 splits
// after
let is_promotion = inputs.iter().any(|s| s.rg_partition_prefix_len != target_prefix_len);
merge_parquet_split_metadata(&inputs, &output, is_promotion)?;

fn prefix_len_ok(inputs: &[ParquetSplitMetadata], mixed_prefix_ok: bool) -> bool {
    mixed_prefix_ok
        || inputs.iter().all(|s| s.rg_partition_prefix_len == inputs[0].rg_partition_prefix_len)
}

fn prefix_homogeneous(inputs: &[ParquetSplitMetadata]) -> Option<u32> {
    let p = inputs.first()?.rg_partition_prefix_len;
    inputs.iter().all(|s| s.rg_partition_prefix_len == p).then_some(p)
}

match merge_parquet_split_metadata(&inputs, &output, mixed_prefix_ok) {
    Err(e) if e.to_string().contains("rg_partition_prefix_len") => {
        warn!("mixed prefix lengths in regular merge: {e:#}; converting to legacy promotion");
        merge_parquet_split_metadata(&inputs, &output, true) // only if this truly is a promotion
    }
    other => other?,
}

When it happens

Trigger: Calling merge_parquet_split_metadata with mixed_prefix_ok=false and inputs with differing rg_partition_prefix_len — e.g. a regular compaction sweep accidentally picks up legacy (prefix-len 0) splits together with prefix-partitioned splits, instead of routing them through the legacy-promotion operation.

Common situations: Upgrading an index that previously had no prefix partitioning: old splits have prefix_len 0 while new ones have the configured length; merge policy written before the prefix-len field existed didn't filter on it; misconfigured legacy-promotion flag passed as false for a promotion batch.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• input {} has kind {:?}, expected {:?}
• input {} has index_uid '{}', expected '{}'
• input {} has partition_id {}, expected {}
• input {} has sort_fields '{}', expected '{}'
• input {} has window {:?}, expected {:?}

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/4bbf2d15a404df34. Report an issue: GitHub.