{"record":{"id":"f5b6d6ee7b9612d6","repo":"quickwit-oss/quickwit","slug":"rg-rg-idx-prefix-column-col-name-is-not-pre","errorCode":null,"errorMessage":"RG {rg_idx}: prefix column '{col_name}' is not present in the file's schema","messagePattern":"RG (.+?): prefix column '(.+?)' is not present in the file's schema","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-parquet-engine/src/storage/inspect.rs","lineNumber":268,"sourceCode":"        _ => bail!(\n            \"rg_partition_prefix_len = {} but {} is missing or empty — cannot verify alignment \\\n             without the sort schema\",\n            report.rg_partition_prefix_len,\n            PARQUET_META_SORT_FIELDS\n        ),\n    };\n\n    let prefix_columns =\n        first_n_sort_field_names(&sort_fields_str, report.rg_partition_prefix_len as usize)?;\n\n    for (rg_idx, rg) in report.row_groups.iter().enumerate() {\n        for col_name in &prefix_columns {\n            let col = rg\n                .columns\n                .iter()\n                .find(|c| c.column_path == *col_name)\n                .ok_or_else(|| {\n                    anyhow::anyhow!(\n                        \"RG {rg_idx}: prefix column '{col_name}' is not present in the file's \\\n                         schema\",\n                    )\n                })?;\n\n            match (&col.chunk_min, &col.chunk_max) {\n                (Some(min), Some(max)) if min == max => {\n                    // OK: column has a single value across the entire RG.\n                }\n                (Some(min), Some(max)) => bail!(\n                    \"RG {rg_idx} violates rg_partition_prefix_len={} claim: column '{col_name}' \\\n                     has min={min:?} != max={max:?} (must be constant across the row group)\",\n                    report.rg_partition_prefix_len\n                ),\n                _ => bail!(\n                    \"RG {rg_idx} violates rg_partition_prefix_len={} claim: column '{col_name}' \\\n                     has no chunk-level statistics\",\n                    report.rg_partition_prefix_len","sourceCodeStart":250,"sourceCodeEnd":286,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-parquet-engine/src/storage/inspect.rs#L250-L286","documentation":"verify_partition_prefix checks every row group (RG) of a parquet file for the prefix columns used in partition pruning, validating chunk_min/chunk_max consistency. This error is raised when a prefix column named in the expected prefix_columns list is not found among the file's row-group column metadata (column_path match fails) for a given RG.","triggerScenarios":"Calling verify_partition_prefix on a parquet file whose schema lacks one of the configured prefix columns — typically after a schema change, an index config edit, or reading a file written by an older/newer writer version.","commonSituations":"Adding/renaming a sort or prefix column in the index config while old data files remain; hand-edited or third-party-written parquet files with an incomplete schema; RG-level schema mismatches in multi-RG files.","solutions":["Check that prefix_columns matches the file's actual schema (parquet-tools / inspect the file metadata)","Rewrite or reindex old data files whose schema predates the prefix column addition","Verify RG-level column_path spelling matches exactly (parquet paths can include nested names)","Skip verification for legacy files if backward compatibility is intended (with care — do not hide failures)"],"exampleFix":"// before: config referencing removed column\nprefix_columns = [\"host\", \"service\", \"region\"]  // file has only host, service\n// after: align config with file schema\nprefix_columns = [\"host\", \"service\"]","handlingStrategy":"validation","validationCode":"let paths: HashSet<_> = rg.columns.iter().map(|c| c.column_path.as_str()).collect();\nfor col in &prefix_columns { ensure!(paths.contains(col), \"missing prefix column {col}\"); }","typeGuard":null,"tryCatchPattern":"match verify_partition_prefix(&file, &prefix_columns) {\n    Err(e) if e.to_string().contains(\"not present in the file's schema\") => { /* legacy file: rewrite or skip */ }\n    Err(e) => return Err(e),\n    Ok(()) => {},\n}","preventionTips":["Keep prefix_columns in the index config in sync with the written schema","Reindex files written before a prefix-column change","Inspect file metadata (parquet-tools) when ingesting third-party parquet"],"tags":["parquet","schema","verification","row-group"],"backgroundTag":"schema-validation-failed","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}