{"record":{"id":"5f40facf7c7a084f","repo":"quickwit-oss/quickwit","slug":"input-file-is-missing-metadata","errorCode":null,"errorMessage":"input file {} is missing {} metadata","messagePattern":"input file (.+?) is missing (.+?) metadata","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-parquet-engine/src/merge/mod.rs","lineNumber":359,"sourceCode":"    for path in paths {\n        let file = std::fs::File::open(path)\n            .with_context(|| format!(\"opening file for metadata: {}\", path.display()))?;\n        let builder = ParquetRecordBatchReaderBuilder::try_new(file)\n            .with_context(|| format!(\"reading footer for metadata: {}\", path.display()))?;\n\n        let kv_metadata = builder.metadata().file_metadata().key_value_metadata();\n\n        let find_kv = |key: &str| -> Option<String> {\n            kv_metadata.and_then(|kvs| {\n                kvs.iter()\n                    .find(|kv| kv.key == key)\n                    .and_then(|kv| kv.value.clone())\n            })\n        };\n\n        // Sort fields: required, must be consistent across all inputs.\n        let file_sort_fields = find_kv(PARQUET_META_SORT_FIELDS).ok_or_else(|| {\n            anyhow::anyhow!(\n                \"input file {} is missing {} metadata\",\n                path.display(),\n                PARQUET_META_SORT_FIELDS\n            )\n        })?;\n\n        match &consensus_sort_fields {\n            Some(expected) => {\n                let expected_schema = parse_sort_fields(expected)?;\n                let file_schema = parse_sort_fields(&file_sort_fields).with_context(|| {\n                    format!(\n                        \"parsing sort schema from {}: '{}'\",\n                        path.display(),\n                        file_sort_fields\n                    )\n                })?;\n                if !equivalent_schemas_for_compaction(&expected_schema, &file_schema) {\n                    bail!(","sourceCodeStart":341,"sourceCodeEnd":377,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-parquet-engine/src/merge/mod.rs#L341-L377","documentation":"Before merging, extract_and_validate_input_metadata reads Quickwit-specific key-value metadata from each parquet file. The sort-fields metadata (PARQUET_META_SORT_FIELDS) is mandatory; when a file lacks it the merge aborts with this error naming the file path and the missing key.","triggerScenarios":"merge_sorted_parquet_files_impl encountering an input parquet file written without the PARQUET_META_SORT_FIELDS key-value entry (e.g. written by vanilla arrow/parquet tooling instead of Quickwit's writer).","commonSituations":"Ingesting externally produced parquet files into the merge pipeline; files written by an older Quickwit version predating this metadata; a writer bug that skipped emitting the metadata.","solutions":["Re-write the file with Quickwit's parquet writer so it embeds the sort-fields metadata.","Verify which file lacks metadata (path is in the message) and exclude it from the merge set if it's foreign input.","Upgrade the producing pipeline to a version that writes PARQUET_META_SORT_FIELDS."],"exampleFix":"// before: writing with generic arrow parquet writer\nlet mut writer = ParquetWriter::try_new(file, schema)?;\n// after: attach required metadata\nlet mut writer = ParquetWriter::try_new_with_options(file, schema, /* kv_metadata: */ Some(vec![(PARQUET_META_SORT_FIELDS, sort_fields_json)]))?;","handlingStrategy":"validation","validationCode":"// before merging, check kv metadata of each parquet file\nlet meta = parquet::file::reader::SerializedFileReader::new(file)?.metadata()\n    .file_metadata().key_value_metadata();\nassert!(meta.iter().flatten().any(|kv| kv.key == PARQUET_META_SORT_FIELDS),\n        \"{} lacks sort-fields metadata\", path.display());","typeGuard":null,"tryCatchPattern":"match merge_sorted_parquet_files(&paths) {\n    Err(e) if e.to_string().contains(\"is missing\") && e.to_string().contains(\"metadata\") => {\n        // re-write the file with Quickwit's writer or exclude it\n    }\n    other => other?,\n}","preventionTips":["Only merge files produced by Quickwit's parquet writer.","Spot-check key-value metadata of externally produced files before ingestion.","Keep producer and consumer Quickwit versions compatible."],"tags":["parquet","metadata","merge","file-format"],"backgroundTag":"missing-required-metadata","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}