quickwit-oss/quickwit · error

legacy multi-RG inputs (rg_partition_prefix_len=0) must go t

Error message

legacy multi-RG inputs (rg_partition_prefix_len=0) must go through the PR-5 adapter — input {idx} has {num_rgs} row groups with no alignment claim

What it means

When inputs carry no alignment claim (rg_partition_prefix_len == 0), the streaming merger assumes each input has at most one row group so rows can be merged positionally. If such a legacy input has multiple row groups, row groups cannot be aligned across inputs, and the merger refuses — directing the caller to the legacy (PR-5) adapter that re-chunks row groups first.

Source

Thrown at quickwit/quickwit-parquet-engine/src/merge/streaming.rs:181

    // AND any input has >1 row group). These have no alignment claim,
    // so RG boundaries are arbitrary row counts that may split a
    // single sort-key value across two RGs. The streaming engine
    // cannot determine merge regions without column-chunk-bounded
    // buffering; such inputs must go through `LegacyInputAdapter`
    // (from PR-5, see `storage::legacy_adapter`), which presents
    // them as one synthetic single-RG stream.
    //
    // This guard catches caller bugs — production code always routes
    // legacy splits through the adapter (see `merge::execute_merge_operation`
    // in `merge/mod.rs`), so a raw legacy `StreamingParquetReader`
    // arriving here is a wiring mistake, not a supported input shape.
    // Bail with a clear pointer rather than wading further into the
    // streaming pipeline with mis-aligned RGs.
    if input_meta.rg_partition_prefix_len == 0 {
        for (idx, stream) in inputs.iter().enumerate() {
            let num_rgs = stream.metadata().num_row_groups();
            if num_rgs > 1 {
                bail!(
                    "legacy multi-RG inputs (rg_partition_prefix_len=0) must go through the PR-5 \
                     adapter — input {idx} has {num_rgs} row groups with no alignment claim"
                );
            }
        }
    }

    info!(
        num_inputs = inputs.len(),
        num_outputs = config.num_outputs,
        sort_fields = %input_meta.sort_fields,
        "starting streaming sorted parquet merge"
    );

    let output_dir = output_dir.to_path_buf();
    let writer_config = config.writer_config.clone();
    let num_outputs = config.num_outputs;

View on GitHub (pinned to a39730c5cd)

Solutions

1. Route legacy multi-row-group inputs through the PR-5 legacy adapter which re-slices row groups into aligned single-RG streams.
2. Re-write legacy splits with the current writer so they carry rg_partition_prefix_len metadata.
3. Update the merge scheduler to detect prefix_len == 0 with num_row_groups > 1 and select the adapter path.
4. If the file is small, rewrite it to a single row group before merging.

Example fix

// before: legacy file straight into streaming merger
let streams = open_streams(&legacy_splits);
streaming_merge_sorted_parquet_files(streams, &out, &cfg).await?;
// after: adapt legacy inputs first
let streams = if needs_legacy_adapter(&legacy_splits) {
    open_adapted_streams(&legacy_splits) // PR-5 adapter re-chunks row groups
} else {
    open_streams(&legacy_splits)
};
streaming_merge_sorted_parquet_files(streams, &out, &cfg).await?;

fn needs_legacy_adapter(streams: &[Box<dyn ColumnPageStream>]) -> bool {
    streams.iter().any(|s| {
        s.metadata().num_row_groups() > 1
            && read_prefix_len(&s.metadata()) == Some(0)
    })
}

match streaming_merge_sorted_parquet_files(streams, &out, &cfg).await {
    Err(e) if e.to_string().contains("PR-5 adapter") => {
        // re-open inputs through the legacy adapter and retry
    }
    other => other,
}

When it happens

Trigger: Calling streaming_merge_sorted_parquet_files with an input whose parquet metadata lacks rg_partition_prefix_len (treated as 0) and whose file contains more than one row group.

Common situations: Merging splits written by old writer versions that produced multi-row-group files without the prefix-length stamp; re-running the streaming merger on legacy data without the adapter.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

• rg_partition_prefix_len mismatch in {}: expected {}, found {
• merge_parquet_split_metadata requires at least one input spl
• input file {} is missing the '{}' column
• sort schema mismatch in {}: expected '{}', found '{}'
• window_start mismatch in {}: expected {:?}, found {:?}

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/aec2b7843109eca1. Report an issue: GitHub.