quickwit-oss/quickwit · error

rg_partition_prefix_len mismatch in {}: expected {}, found {

Error message

rg_partition_prefix_len mismatch in {}: expected {}, found {} — splits with different prefix lengths must not appear in the same merge

What it means

All merge inputs must have been produced with the same row-group partition prefix length (`rg_partition_prefix_len`). If a file's metadata value differs from the consensus (missing defaults to 0), the merge is aborted because row groups with different prefix lengths are not guaranteed to be aligned and cannot be merged positionally.

Source

Thrown at quickwit/quickwit-parquet-engine/src/merge/mod.rs:465

            .with_context(|| format!("parsing num_merge_ops from {}", path.display()))?
            .unwrap_or(0);

        if file_merge_ops > max_merge_ops {
            max_merge_ops = file_merge_ops;
        }

        // Row group partition prefix length: must be consistent across all
        // inputs. Absent KV → 0 (legacy default; no alignment claim).
        let file_prefix_len = find_kv(PARQUET_META_RG_PARTITION_PREFIX_LEN)
            .map(|s| s.parse::<u32>())
            .transpose()
            .with_context(|| format!("parsing rg_partition_prefix_len from {}", path.display()))?
            .unwrap_or(0);

        match consensus_prefix_len {
            Some(expected) => {
                if file_prefix_len != expected {
                    bail!(
                        "rg_partition_prefix_len mismatch in {}: expected {}, found {} — splits \
                         with different prefix lengths must not appear in the same merge",
                        path.display(),
                        expected,
                        file_prefix_len
                    );
                }
            }
            None => {
                consensus_prefix_len = Some(file_prefix_len);
            }
        }
    }

    Ok(InputMetadata {
        sort_fields: consensus_sort_fields.expect("at least one input required"),
        window_start_secs: consensus_window_start.expect("at least one input required"),
        window_duration_secs: consensus_window_duration.unwrap_or(0),

View on GitHub (pinned to a39730c5cd)

Solutions

1. Ensure the merge operation only contains splits sharing the same rg_partition_prefix_len value.
2. Route legacy multi-row-group files (prefix_len 0) through the legacy/PR-5 adapter path instead of the streaming merger.
3. Re-write old splits through the current writer so they are stamped with the current prefix length.
4. Fix the control plane / merge scheduler to include prefix_len in its split-grouping key.

Example fix

// before: mixing legacy and aligned splits
execute_merge_operation(&op, load(op.splits), &out, &cfg).await?;
// after: assert homogeneity before scheduling
let lens: HashSet<_> = op.splits.iter().map(|s| s.rg_partition_prefix_len).collect();
anyhow::ensure!(lens.len() == 1, "mixed prefix lengths in merge task");
execute_merge_operation(&op, load(op.splits), &out, &cfg).await?;

fn prefix_lens_consistent(splits: &[Split]) -> anyhow::Result<()> {
    let lens: HashSet<u32> = splits.iter().map(|s| s.rg_partition_prefix_len).collect();
    anyhow::ensure!(lens.len() == 1, "splits with different rg_partition_prefix_len grouped together");
    Ok(())
}

match execute_merge_operation(...).await {
    Err(e) if e.to_string().contains("rg_partition_prefix_len mismatch") => {
        // split the merge task into per-prefix-len groups and retry each
    }
    other => other?,
}

When it happens

Trigger: Calling merge_sorted_parquet_files_impl with files whose `rg_partition_prefix_len` KV metadata values disagree, e.g. mixing legacy files (key absent, 0) with PR-5-style aligned files.

Common situations: Upgrading Quickwit and merging pre-upgrade splits with post-upgrade ones; a compaction scheduler that does not filter splits by prefix length; manually crafted merge tasks including heterogeneous splits.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• legacy multi-RG inputs (rg_partition_prefix_len=0) must go t
• merge_parquet_split_metadata requires at least one input spl
• input file {} is missing the '{}' column
• sort schema mismatch in {}: expected '{}', found '{}'
• window_start mismatch in {}: expected {:?}, found {:?}

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/32c364ab120f928b. Report an issue: GitHub.