{"record":{"id":"05f2ea1fb4fdca6e","repo":"vectordotdev/vector","slug":"parquet","errorCode":null,"errorMessage":"{}","messagePattern":"\\{\\}","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"lib/codecs/src/encoding/format/parquet.rs","lineNumber":382,"sourceCode":"        }\n\n        let record_batch =\n            build_record_batch(Arc::clone(&self.schema), &json_values).map_err(Box::new)?;\n\n        Self::write_record_batch(&record_batch, buffer, &self.writer_props).map_err(Box::new)?;\n\n        Ok(())\n    }\n}\n\npub struct ParquetSchemaGenerator {}\n\nimpl ParquetSchemaGenerator {\n    pub fn infer_schema(events: &[serde_json::Value]) -> Result<Schema, Error> {\n        let schema = infer_json_schema_from_iterator(events.iter().map(Ok::<_, ArrowError>))\n            .map_err(|e| {\n                emit(SchemaGenerationError { error: &e });\n                Error::new(ErrorKind::InvalidData, e.to_string())\n            })?;\n\n        Ok(schema)\n    }\n\n    /// Attempt to modify schema to set timestamp fields as Timestamp instead of Utf8.\n    /// Only works for top-level fields.\n    fn try_normalize_schema(events: &[Event], schema: Schema) -> Schema {\n        let mut ts_seen: HashSet<String> = HashSet::new();\n        let mut non_ts_seen: HashSet<String> = HashSet::new();\n\n        for event in events.iter().filter_map(Event::maybe_as_log) {\n            if let Some(object_map) = event.as_map() {\n                for (path, value) in object_map {\n                    if value.is_timestamp() {\n                        ts_seen.insert(path.to_string());\n                    } else if !value.is_null() {\n                        non_ts_seen.insert(path.to_string());","sourceCodeStart":364,"sourceCodeEnd":400,"githubUrl":"https://github.com/vectordotdev/vector/blob/bdb87aeaa4c4ff27c0ba643c1c77b21bf2ef4013/lib/codecs/src/encoding/format/parquet.rs#L364-L400","documentation":"The Parquet schema generator infers an Arrow schema from a batch of JSON events using the JSON schema inference utility. This error wraps any error returned by that inference (e.g. conflicting types for the same field across events) into an io::Error, with the message being the underlying error's display text (the \"{}\" template). A SchemaGenerationError event is emitted before returning.","triggerScenarios":"Calling infer_schema with events whose inferred JSON schema conflicts — e.g. one event has field 'count' as a number and another has it as a string, or nested structures disagree on types.","commonSituations":"Heterogeneous log data feeding a parquet sink without type normalization; schema drift after a producer update; optional fields that sometimes appear as null and sometimes as values.","solutions":["Normalize field types across events before inference (coerce to a single type per field).","Pre-validate the event batch for type conflicts and drop or cast conflicting records.","Increase sampling/handling so nulls are handled and only mutually compatible events are used for schema inference."],"exampleFix":"// before\n[{\"count\": 1}, {\"count\": \"many\"}]\n// after\n[{\"count\": 1}, {\"count\": 2}]  // or coerce all to strings","handlingStrategy":"try-catch","validationCode":"// detect type conflicts in events before schema inference\nfn has_conflicting_types(events: &[serde_json::Value], key: &str) -> bool {\n    events.iter().filter_map(|e| e.get(key))\n        .map(|v| match v { serde_json::Value::Number(_) => \"num\", serde_json::Value::String(_) => \"str\", _ => \"other\" })\n        .collect::<std::collections::HashSet<_>>().len() > 1\n}","typeGuard":null,"tryCatchPattern":"match generator.infer_schema(&events) {\n    Ok(schema) => schema,\n    Err(e) => {\n        tracing::warn!(\"schema inference failed: {e}; normalizing types and retrying\");\n        let normalized = coerce_event_types(&events);\n        generator.infer_schema(&normalized)?\n    }\n}","preventionTips":["Coerce numeric-like strings to numbers (or vice versa) upstream.","Filter out null/empty events before inference.","Pin producer field types with a schema contract."],"tags":["parquet","schema","arrow","encoding"],"backgroundTag":"schema-validation-failed","analyzedSha":"bdb87aeaa4c4ff27c0ba643c1c77b21bf2ef4013","analyzedAt":"2026-09-16T02:53:35.741Z","contentChangedAt":"2026-09-16T02:53:35.741Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}