quickwit-oss/quickwit · error

input file {} is missing the '{}' column

Error message

input file {} is missing the '{}' column

What it means

read_inputs verifies that each merged Parquet file contains the special `sorted_series` column (SORTED_SERIES_COLUMN) after its batches are concatenated. This bail fires when a file's schema lacks that column, meaning the file was not written by this engine's sorted-writer and cannot take part in the sorted merge, which orders rows by that column.

Source

Thrown at quickwit/quickwit-parquet-engine/src/merge/mod.rs:315

                let f = std::fs::File::open(path)?;
                let b = ParquetRecordBatchReaderBuilder::try_new(f)?;
                b.schema().clone()
            };
            batches.push(RecordBatch::new_empty(schema));
            continue;
        }

        let schema = file_batches[0].schema();
        let concatenated = arrow::compute::concat_batches(&schema, &file_batches)
            .with_context(|| format!("concatenating batches: {}", path.display()))?;

        // Verify sorted_series column exists.
        if concatenated
            .schema()
            .index_of(SORTED_SERIES_COLUMN)
            .is_err()
        {
            bail!(
                "input file {} is missing the '{}' column",
                path.display(),
                SORTED_SERIES_COLUMN
            );
        }

        batches.push(concatenated);
    }

    Ok(batches)
}

/// Extract and validate metadata from all input files.
///
/// Reads `qh.*` keys from each file's Parquet KV metadata. Validates that
/// all inputs share the same sort schema (via `equivalent_schemas_for_compaction`),
/// window_start, and window_duration. Returns the consensus metadata plus
/// `max(num_merge_ops) + 1` for the output.

View on GitHub (pinned to a39730c5cd)

Solutions

  1. Filter input paths to files that carry this engine's Parquet KV metadata before calling the merge (check for the `qh.*` keys).
  2. Exclude legacy-format files and route them through a separate migration/rewrite path first.
  3. Fix the directory listing/glob so only engine-produced split files are selected.
  4. Verify file provenance (KV metadata / split id filename pattern) to identify which file is foreign.

Example fix

// before
let paths: Vec<PathBuf> = storage.list(dir)?.collect();
merge_sorted_parquet_files(&paths, out, &config)?;
// after
let paths: Vec<PathBuf> = storage.list(dir)?
    .filter(|p| has_engine_kv_metadata(p)) // requires sorted_series column
    .collect();
merge_sorted_parquet_files(&paths, out, &config)?;
Defensive patterns

Strategy: validation

Validate before calling

fn has_sorted_series_column(path: &Path) -> bool {
    let file = std::fs::File::open(path).expect("open");
    let builder = ParquetRecordBatchReaderBuilder::try_new(file).expect("footer");
    builder.schema().column_with_name(SORTED_SERIES_COLUMN).is_some()
}
let paths: Vec<_> = paths.into_iter().filter(|p| has_sorted_series_column(p)).collect();

Type guard

fn is_engine_file(path: &Path) -> bool {
    ParquetRecordBatchReaderBuilder::try_new(std::fs::File::open(path).ok()?)
        .ok()?
        .schema()
        .column_with_name(SORTED_SERIES_COLUMN)
        .is_some()
}

Try / catch

match merge_sorted_parquet_files(&paths, out_dir, &config) {
    Err(e) if e.to_string().contains("is missing the") => {
        error!("non-engine parquet file in merge batch: {e:#}; excluding and re-running");
        let filtered: Vec<_> = paths.into_iter().filter(|p| is_engine_file(p)).collect();
        merge_sorted_parquet_files(&filtered, out_dir, &config)
    }
    other => other,
}

Prevention

When it happens

Trigger: Calling merge_sorted_parquet_files with a path pointing at a plain/foreign Parquet file (no qh.* KV metadata, no sorted_series column) — e.g. an externally produced Parquet file, a file written by an older engine version before the column existed, or a wrong path passed in the input list.

Common situations: Mixing files from a legacy format migration into a compaction batch; pointing the merge at a debug/export dump; a storage listing that includes non-engine files (e.g. .parquet uploads from another tool) in the same directory.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/e0cde1461aba24c7. Report an issue: GitHub.