quickwit-oss/quickwit · error

input file is missing metadata

Error message

input file {} is missing {} metadata

What it means

Before merging, extract_and_validate_input_metadata reads Quickwit-specific key-value metadata from each parquet file. The sort-fields metadata (PARQUET_META_SORT_FIELDS) is mandatory; when a file lacks it the merge aborts with this error naming the file path and the missing key.

Solutions

  1. Re-write the file with Quickwit's parquet writer so it embeds the sort-fields metadata.
  2. Verify which file lacks metadata (path is in the message) and exclude it from the merge set if it's foreign input.
  3. Upgrade the producing pipeline to a version that writes PARQUET_META_SORT_FIELDS.

Example fix

// before: writing with generic arrow parquet writer
let mut writer = ParquetWriter::try_new(file, schema)?;
// after: attach required metadata
let mut writer = ParquetWriter::try_new_with_options(file, schema, /* kv_metadata: */ Some(vec![(PARQUET_META_SORT_FIELDS, sort_fields_json)]))?;
Defensive patterns

Strategy: validation

Validate before calling

// before merging, check kv metadata of each parquet file
let meta = parquet::file::reader::SerializedFileReader::new(file)?.metadata()
    .file_metadata().key_value_metadata();
assert!(meta.iter().flatten().any(|kv| kv.key == PARQUET_META_SORT_FIELDS),
        "{} lacks sort-fields metadata", path.display());

Try / catch

match merge_sorted_parquet_files(&paths) {
    Err(e) if e.to_string().contains("is missing") && e.to_string().contains("metadata") => {
        // re-write the file with Quickwit's writer or exclude it
    }
    other => other?,
}

Prevention

When it happens

Trigger: merge_sorted_parquet_files_impl encountering an input parquet file written without the PARQUET_META_SORT_FIELDS key-value entry (e.g. written by vanilla arrow/parquet tooling instead of Quickwit's writer).

Common situations: Ingesting externally produced parquet files into the merge pipeline; files written by an older Quickwit version predating this metadata; a writer bug that skipped emitting the metadata.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/5f40facf7c7a084f. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-parquet-engine/src/merge/mod.rs:359

    for path in paths {
        let file = std::fs::File::open(path)
            .with_context(|| format!("opening file for metadata: {}", path.display()))?;
        let builder = ParquetRecordBatchReaderBuilder::try_new(file)
            .with_context(|| format!("reading footer for metadata: {}", path.display()))?;

        let kv_metadata = builder.metadata().file_metadata().key_value_metadata();

        let find_kv = |key: &str| -> Option<String> {
            kv_metadata.and_then(|kvs| {
                kvs.iter()
                    .find(|kv| kv.key == key)
                    .and_then(|kv| kv.value.clone())
            })
        };

        // Sort fields: required, must be consistent across all inputs.
        let file_sort_fields = find_kv(PARQUET_META_SORT_FIELDS).ok_or_else(|| {
            anyhow::anyhow!(
                "input file {} is missing {} metadata",
                path.display(),
                PARQUET_META_SORT_FIELDS
            )
        })?;

        match &consensus_sort_fields {
            Some(expected) => {
                let expected_schema = parse_sort_fields(expected)?;
                let file_schema = parse_sort_fields(&file_sort_fields).with_context(|| {
                    format!(
                        "parsing sort schema from {}: '{}'",
                        path.display(),
                        file_sort_fields
                    )
                })?;
                if !equivalent_schemas_for_compaction(&expected_schema, &file_schema) {
                    bail!(

View on GitHub (pinned to a39730c5cd)