{"record":{"id":"aec2b7843109eca1","repo":"quickwit-oss/quickwit","slug":"legacy-multi-rg-inputs-rg-partition-prefix-len-0","errorCode":null,"errorMessage":"legacy multi-RG inputs (rg_partition_prefix_len=0) must go through the PR-5 adapter — input {idx} has {num_rgs} row groups with no alignment claim","messagePattern":"legacy multi-RG inputs \\(rg_partition_prefix_len=0\\) must go through the PR-5 adapter — input (.+?) has (.+?) row groups with no alignment claim","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-parquet-engine/src/merge/streaming.rs","lineNumber":181,"sourceCode":"    // AND any input has >1 row group). These have no alignment claim,\n    // so RG boundaries are arbitrary row counts that may split a\n    // single sort-key value across two RGs. The streaming engine\n    // cannot determine merge regions without column-chunk-bounded\n    // buffering; such inputs must go through `LegacyInputAdapter`\n    // (from PR-5, see `storage::legacy_adapter`), which presents\n    // them as one synthetic single-RG stream.\n    //\n    // This guard catches caller bugs — production code always routes\n    // legacy splits through the adapter (see `merge::execute_merge_operation`\n    // in `merge/mod.rs`), so a raw legacy `StreamingParquetReader`\n    // arriving here is a wiring mistake, not a supported input shape.\n    // Bail with a clear pointer rather than wading further into the\n    // streaming pipeline with mis-aligned RGs.\n    if input_meta.rg_partition_prefix_len == 0 {\n        for (idx, stream) in inputs.iter().enumerate() {\n            let num_rgs = stream.metadata().num_row_groups();\n            if num_rgs > 1 {\n                bail!(\n                    \"legacy multi-RG inputs (rg_partition_prefix_len=0) must go through the PR-5 \\\n                     adapter — input {idx} has {num_rgs} row groups with no alignment claim\"\n                );\n            }\n        }\n    }\n\n    info!(\n        num_inputs = inputs.len(),\n        num_outputs = config.num_outputs,\n        sort_fields = %input_meta.sort_fields,\n        \"starting streaming sorted parquet merge\"\n    );\n\n    let output_dir = output_dir.to_path_buf();\n    let writer_config = config.writer_config.clone();\n    let num_outputs = config.num_outputs;\n","sourceCodeStart":163,"sourceCodeEnd":199,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-parquet-engine/src/merge/streaming.rs#L163-L199","documentation":"When inputs carry no alignment claim (rg_partition_prefix_len == 0), the streaming merger assumes each input has at most one row group so rows can be merged positionally. If such a legacy input has multiple row groups, row groups cannot be aligned across inputs, and the merger refuses — directing the caller to the legacy (PR-5) adapter that re-chunks row groups first.","triggerScenarios":"Calling streaming_merge_sorted_parquet_files with an input whose parquet metadata lacks rg_partition_prefix_len (treated as 0) and whose file contains more than one row group.","commonSituations":"Merging splits written by old writer versions that produced multi-row-group files without the prefix-length stamp; re-running the streaming merger on legacy data without the adapter.","solutions":["Route legacy multi-row-group inputs through the PR-5 legacy adapter which re-slices row groups into aligned single-RG streams.","Re-write legacy splits with the current writer so they carry rg_partition_prefix_len metadata.","Update the merge scheduler to detect prefix_len == 0 with num_row_groups > 1 and select the adapter path.","If the file is small, rewrite it to a single row group before merging."],"exampleFix":"// before: legacy file straight into streaming merger\nlet streams = open_streams(&legacy_splits);\nstreaming_merge_sorted_parquet_files(streams, &out, &cfg).await?;\n// after: adapt legacy inputs first\nlet streams = if needs_legacy_adapter(&legacy_splits) {\n    open_adapted_streams(&legacy_splits) // PR-5 adapter re-chunks row groups\n} else {\n    open_streams(&legacy_splits)\n};\nstreaming_merge_sorted_parquet_files(streams, &out, &cfg).await?;","handlingStrategy":"validation","validationCode":"fn needs_legacy_adapter(streams: &[Box<dyn ColumnPageStream>]) -> bool {\n    streams.iter().any(|s| {\n        s.metadata().num_row_groups() > 1\n            && read_prefix_len(&s.metadata()) == Some(0)\n    })\n}","typeGuard":null,"tryCatchPattern":"match streaming_merge_sorted_parquet_files(streams, &out, &cfg).await {\n    Err(e) if e.to_string().contains(\"PR-5 adapter\") => {\n        // re-open inputs through the legacy adapter and retry\n    }\n    other => other,\n}","preventionTips":["Check num_row_groups and prefix_len metadata when selecting the merge path.","Rewrite legacy splits with the current writer during upgrades.","Keep the legacy adapter wired into the merge scheduler for old data."],"tags":["merge","parquet","row-group","legacy-compat"],"backgroundTag":"incompatible-source-type","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}