quickwit-oss/quickwit · error

input file is missing the ' ' column

Error message

input file {} is missing the '{}' column

What it means

read_inputs verifies that each merged Parquet file contains the special `sorted_series` column (SORTED_SERIES_COLUMN) after its batches are concatenated. This bail fires when a file's schema lacks that column, meaning the file was not written by this engine's sorted-writer and cannot take part in the sorted merge, which orders rows by that column.

Solutions

1. Filter input paths to files that carry this engine's Parquet KV metadata before calling the merge (check for the `qh.*` keys).
2. Exclude legacy-format files and route them through a separate migration/rewrite path first.
3. Fix the directory listing/glob so only engine-produced split files are selected.
4. Verify file provenance (KV metadata / split id filename pattern) to identify which file is foreign.

Example fix

// before
let paths: Vec<PathBuf> = storage.list(dir)?.collect();
merge_sorted_parquet_files(&paths, out, &config)?;
// after
let paths: Vec<PathBuf> = storage.list(dir)?
    .filter(|p| has_engine_kv_metadata(p)) // requires sorted_series column
    .collect();
merge_sorted_parquet_files(&paths, out, &config)?;

fn has_sorted_series_column(path: &Path) -> bool {
    let file = std::fs::File::open(path).expect("open");
    let builder = ParquetRecordBatchReaderBuilder::try_new(file).expect("footer");
    builder.schema().column_with_name(SORTED_SERIES_COLUMN).is_some()
}
let paths: Vec<_> = paths.into_iter().filter(|p| has_sorted_series_column(p)).collect();

fn is_engine_file(path: &Path) -> bool {
    ParquetRecordBatchReaderBuilder::try_new(std::fs::File::open(path).ok()?)
        .ok()?
        .schema()
        .column_with_name(SORTED_SERIES_COLUMN)
        .is_some()
}

match merge_sorted_parquet_files(&paths, out_dir, &config) {
    Err(e) if e.to_string().contains("is missing the") => {
        error!("non-engine parquet file in merge batch: {e:#}; excluding and re-running");
        let filtered: Vec<_> = paths.into_iter().filter(|p| is_engine_file(p)).collect();
        merge_sorted_parquet_files(&filtered, out_dir, &config)
    }
    other => other,
}

When it happens

Trigger: Calling merge_sorted_parquet_files with a path pointing at a plain/foreign Parquet file (no qh.* KV metadata, no sorted_series column) — e.g. an externally produced Parquet file, a file written by an older engine version before the column existed, or a wrong path passed in the input list.

Common situations: Mixing files from a legacy format migration into a compaction batch; pointing the merge at a debug/export dump; a storage listing that includes non-engine files (e.g. .parquet uploads from another tool) in the same directory.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• input is missing column
• sort schema ' ' does not contain a timestamp column
• sort schema mismatch in
• timeseries_id column is required in the batch for…
• concatenate type must have at least one sub-field

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/e0cde1461aba24c7. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-parquet-engine/src/merge/mod.rs:315

                let f = std::fs::File::open(path)?;
                let b = ParquetRecordBatchReaderBuilder::try_new(f)?;
                b.schema().clone()
            };
            batches.push(RecordBatch::new_empty(schema));
            continue;
        }

        let schema = file_batches[0].schema();
        let concatenated = arrow::compute::concat_batches(&schema, &file_batches)
            .with_context(|| format!("concatenating batches: {}", path.display()))?;

        // Verify sorted_series column exists.
        if concatenated
            .schema()
            .index_of(SORTED_SERIES_COLUMN)
            .is_err()
        {
            bail!(
                "input file {} is missing the '{}' column",
                path.display(),
                SORTED_SERIES_COLUMN
            );
        }

        batches.push(concatenated);
    }

    Ok(batches)
}

/// Extract and validate metadata from all input files.
///
/// Reads `qh.*` keys from each file's Parquet KV metadata. Validates that
/// all inputs share the same sort schema (via `equivalent_schemas_for_compaction`),
/// window_start, and window_duration. Returns the consensus metadata plus
/// `max(num_merge_ops) + 1` for the output.

View on GitHub (pinned to a39730c5cd)