{"record":{"id":"e0cde1461aba24c7","repo":"quickwit-oss/quickwit","slug":"input-file-is-missing-the-column","errorCode":null,"errorMessage":"input file {} is missing the '{}' column","messagePattern":"input file (.+?) is missing the '(.+?)' column","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-parquet-engine/src/merge/mod.rs","lineNumber":315,"sourceCode":"                let f = std::fs::File::open(path)?;\n                let b = ParquetRecordBatchReaderBuilder::try_new(f)?;\n                b.schema().clone()\n            };\n            batches.push(RecordBatch::new_empty(schema));\n            continue;\n        }\n\n        let schema = file_batches[0].schema();\n        let concatenated = arrow::compute::concat_batches(&schema, &file_batches)\n            .with_context(|| format!(\"concatenating batches: {}\", path.display()))?;\n\n        // Verify sorted_series column exists.\n        if concatenated\n            .schema()\n            .index_of(SORTED_SERIES_COLUMN)\n            .is_err()\n        {\n            bail!(\n                \"input file {} is missing the '{}' column\",\n                path.display(),\n                SORTED_SERIES_COLUMN\n            );\n        }\n\n        batches.push(concatenated);\n    }\n\n    Ok(batches)\n}\n\n/// Extract and validate metadata from all input files.\n///\n/// Reads `qh.*` keys from each file's Parquet KV metadata. Validates that\n/// all inputs share the same sort schema (via `equivalent_schemas_for_compaction`),\n/// window_start, and window_duration. Returns the consensus metadata plus\n/// `max(num_merge_ops) + 1` for the output.","sourceCodeStart":297,"sourceCodeEnd":333,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-parquet-engine/src/merge/mod.rs#L297-L333","documentation":"read_inputs verifies that each merged Parquet file contains the special `sorted_series` column (SORTED_SERIES_COLUMN) after its batches are concatenated. This bail fires when a file's schema lacks that column, meaning the file was not written by this engine's sorted-writer and cannot take part in the sorted merge, which orders rows by that column.","triggerScenarios":"Calling merge_sorted_parquet_files with a path pointing at a plain/foreign Parquet file (no qh.* KV metadata, no sorted_series column) — e.g. an externally produced Parquet file, a file written by an older engine version before the column existed, or a wrong path passed in the input list.","commonSituations":"Mixing files from a legacy format migration into a compaction batch; pointing the merge at a debug/export dump; a storage listing that includes non-engine files (e.g. .parquet uploads from another tool) in the same directory.","solutions":["Filter input paths to files that carry this engine's Parquet KV metadata before calling the merge (check for the `qh.*` keys).","Exclude legacy-format files and route them through a separate migration/rewrite path first.","Fix the directory listing/glob so only engine-produced split files are selected.","Verify file provenance (KV metadata / split id filename pattern) to identify which file is foreign."],"exampleFix":"// before\nlet paths: Vec<PathBuf> = storage.list(dir)?.collect();\nmerge_sorted_parquet_files(&paths, out, &config)?;\n// after\nlet paths: Vec<PathBuf> = storage.list(dir)?\n    .filter(|p| has_engine_kv_metadata(p)) // requires sorted_series column\n    .collect();\nmerge_sorted_parquet_files(&paths, out, &config)?;","handlingStrategy":"validation","validationCode":"fn has_sorted_series_column(path: &Path) -> bool {\n    let file = std::fs::File::open(path).expect(\"open\");\n    let builder = ParquetRecordBatchReaderBuilder::try_new(file).expect(\"footer\");\n    builder.schema().column_with_name(SORTED_SERIES_COLUMN).is_some()\n}\nlet paths: Vec<_> = paths.into_iter().filter(|p| has_sorted_series_column(p)).collect();","typeGuard":"fn is_engine_file(path: &Path) -> bool {\n    ParquetRecordBatchReaderBuilder::try_new(std::fs::File::open(path).ok()?)\n        .ok()?\n        .schema()\n        .column_with_name(SORTED_SERIES_COLUMN)\n        .is_some()\n}","tryCatchPattern":"match merge_sorted_parquet_files(&paths, out_dir, &config) {\n    Err(e) if e.to_string().contains(\"is missing the\") => {\n        error!(\"non-engine parquet file in merge batch: {e:#}; excluding and re-running\");\n        let filtered: Vec<_> = paths.into_iter().filter(|p| is_engine_file(p)).collect();\n        merge_sorted_parquet_files(&filtered, out_dir, &config)\n    }\n    other => other,\n}","preventionTips":["Pre-screen input files for the sorted_series column / qh.* KV metadata before merging.","Keep engine split files in dedicated directories so foreign parquet files can't be listed.","Route legacy-format files through a rewrite/migration before compaction."],"tags":["rust","merge","parquet","schema","missing-column"],"backgroundTag":"schema-validation-failed","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}