{"record":{"id":"99b9e9f42195ceb1","repo":"quickwit-oss/quickwit","slug":"input-input-idx-rg-rg-idx-col-has-no-stat","errorCode":null,"errorMessage":"input {input_idx} rg {rg_idx} col '{}' has no statistics — cannot determine prefix alignment without min/max","messagePattern":"input (.+?) rg (.+?) col '(.+?)' has no statistics — cannot determine prefix alignment without min/max","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-parquet-engine/src/merge/streaming/region_grouping.rs","lineNumber":241,"sourceCode":"    metadata: &ParquetMetaData,\n    rg_idx: usize,\n    prefix_cols: &[Option<PrefixColumn>],\n    input_idx: usize,\n) -> Result<Vec<u8>> {\n    let rg_meta = metadata.row_group(rg_idx);\n    let mut key = Vec::new();\n    for col_opt in prefix_cols {\n        let Some(col) = col_opt else {\n            // SS-3 implicit null: column absent from schema, so every\n            // row's value is null. Skip the slot entirely — the\n            // trailing prefix-length sentinel will keep this from\n            // colliding with present-value keys, and sorted_series\n            // applies the same \"skip null cols\" rule at the row level.\n            continue;\n        };\n        let chunk = rg_meta.column(col.parquet_col_idx);\n        let stats = chunk.statistics().ok_or_else(|| {\n            anyhow!(\n                \"input {input_idx} rg {rg_idx} col '{}' has no statistics — cannot determine \\\n                 prefix alignment without min/max\",\n                col.name,\n            )\n        })?;\n\n        // Parquet's `num_values` is total cell count including nulls.\n        // `null_count_opt()` returns the explicitly-recorded null\n        // count (defaulting to 0 when absent, per parquet-rs guidance).\n        let num_values = chunk.num_values().max(0) as u64;\n        let null_count = stats.null_count_opt().unwrap_or(0);\n\n        if num_values > 0 && null_count == num_values {\n            // All-null RG: skip the column entirely (don't write its\n            // ordinal byte or value). The trailing prefix-length\n            // sentinel below ensures the resulting all-null key\n            // still sorts after any non-null key. See the sentinel\n            // comment for the full argument.","sourceCodeStart":223,"sourceCodeEnd":259,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-parquet-engine/src/merge/streaming/region_grouping.rs#L223-L259","documentation":"extract_rg_composite_prefix_key builds a byte key for each row group from the min/max column statistics; it requires statistics on every prefix column chunk to verify min==max. When parquet::file::metadata returns None statistics for a chunk, the function cannot prove prefix alignment and raises this error instead of guessing.","triggerScenarios":"Calling extract_regions_from_metadata (or streaming_merge_sorted_parquet_files) on a parquet file whose row-group column chunks were written without statistics — e.g. written with set_statistics disabled or by a writer that omits stats for that column.","commonSituations":"Files produced by external/older writers with statistics turned off; columns where the writer intentionally skipped stats; synthetic test files built without stats.","solutions":["Rewrite/normalize the input parquet files so column chunks carry min/max statistics (re-write with statistics enabled) before the streaming merge.","If the writer is in this repo, ensure statistics are not disabled when writing sorted series inputs.","As a last resort, compute the RG value bounds by decoding the column chunk, but prefer fixing the writer — stats are required by the design."],"exampleFix":"// before: feeding externally-written parquet without stats into streaming merge\n// after: normalize input files first, e.g.\nlet props = WriterProperties::builder().set_statistics_enabled(EnabledStatistics::Page).build();\n// and rewrite the file before calling streaming_merge_sorted_parquet_files","handlingStrategy":"validation","validationCode":"for rg in 0..meta.num_row_groups() {\n    for col_idx in prefix_col_indices {\n        if meta.row_group(rg).column(col_idx).statistics().is_none() {\n            return Err(anyhow!(\"input file missing statistics on rg {rg} col {col_idx}\"));\n        }\n    }\n}\n","typeGuard":null,"tryCatchPattern":"// inspect and recover\nmatch err.downcast_ref::<String>() {\n    Some(msg) if msg.contains(\"has no statistics\") => rewrite_file_with_stats(path)?,\n    _ => return Err(err),\n}\n","preventionTips":["Write all merge-input parquet files with statistics enabled (default writer properties).","Pre-validate inputs with a metadata-only scan before starting a merge job.","Reject externally produced files lacking stats at ingestion time rather than merge time."],"tags":["parquet","statistics","merge","metadata"],"backgroundTag":"missing-required-argument","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}