{"record":{"id":"110923d508d2a9f7","repo":"dbt-labs/dbt-core","slug":"failed-to-serialize-some-records-errors","errorCode":null,"errorMessage":"Failed to serialize some records: {errors}","messagePattern":"Failed to serialize some records: (.+?)","errorType":"error_code","errorClass":"std::io::Error (InvalidData)","httpStatus":null,"severity":"error","filePath":"crates/dbt-tracing/src/serialize/arrow.rs","lineNumber":617,"sourceCode":"        records\n            .iter()\n            .filter(|r| {\n                // Only include records with serializable attributes\n                r.attributes()\n                    .output_flags()\n                    .contains(TelemetryOutputFlags::EXPORT_PARQUET)\n            })\n            .filter_map(|r| {\n                ArrowTelemetryRecord::try_from(r)\n                    .map_err(|e| errors.push(e))\n                    .ok()\n            })\n            .collect();\n\n    if !errors.is_empty() {\n        // As of today, this should never happen because we filter out records with non-serializable attributes\n        // above via export flags and this is the only realistic error case.\n        return Err(Box::new(std::io::Error::new(\n            std::io::ErrorKind::InvalidData,\n            format!(\"Failed to serialize some records: {}\", errors.join(\"; \")),\n        )));\n    }\n\n    // Serialize with the temporary schema (timestamp fields as u64),\n    // see `create_arrow_schema` for details.\n    let batch = serde_arrow::to_record_batch(schemas.serialisable_schema(), &arrow_records)?;\n\n    let mut columns = batch.columns().to_vec();\n\n    // Convert timestamp columns from u64 to Timestamp(NANOSECOND),\n    // this is zero-copy, just metadata change.\n    let schema_with_timestamps = schemas.schema_with_timestamps();\n    for (i, field) in schema_with_timestamps.iter().enumerate() {\n        if let DataType::Timestamp(TimeUnit::Nanosecond, None) = field.data_type()\n            && let Some(column) = columns.get(i)\n        {","sourceCodeStart":599,"sourceCodeEnd":635,"githubUrl":"https://github.com/dbt-labs/dbt-core/blob/0267ce9170576975b76b64ce856b2e5848e96617/crates/dbt-tracing/src/serialize/arrow.rs#L599-L635","documentation":"serialize_to_arrow collects per-record serialization errors into `errors`; if any remain after processing, it returns an std::io::Error with ErrorKind::InvalidData. Per the comment, this is currently treated as an internal invariant: records with non-serializable attributes should already have been filtered out upstream via export flags, so hitting this indicates either a bug in that filtering or unexpected attribute values.","triggerScenarios":"Any record in the batch produces a serialization failure after the export-flag filtering pass — e.g. an attribute value the arrow serializer cannot encode slipping through the filter, or a regression in `flush_batch`'s input path.","commonSituations":"Telemetry/export pipelines flushing batches containing records with unexpected attribute types or values; custom record types added without updating the filter and the serializer; test-roundtrip code feeding synthetic records directly.","solutions":["Inspect the joined `errors` message to find the failing record and attribute","Fix the export-flag filtering upstream so non-serializable records are excluded before serialize_to_arrow","If a new attribute type is legitimately needed, add support for it in the arrow serializer in crates/dbt-tracing/src/serialize/arrow.rs"],"exampleFix":"// before\nlet batch: Vec<_> = records.into_iter().collect();\n// after: filter non-serializable records before serializing\nlet batch: Vec<_> = records\n    .into_iter()\n    .filter(|r| r.has_exportable_attributes())\n    .collect();","handlingStrategy":"validation","validationCode":"let non_exportable: Vec<_> = records.iter()\n    .filter(|r| !r.has_exportable_attributes())\n    .collect();\nif !non_exportable.is_empty() {\n    // drop or fix these records before calling serialize_to_arrow\n}","typeGuard":null,"tryCatchPattern":"match serialize_to_arrow(records) {\n    Err(e) if e.to_string().starts_with(\"Failed to serialize some records\") => {\n        log::error!(\"arrow serialization failed: {e}\");\n        // inspect which record/attribute failed and re-export with filters\n    }\n    other => other?,\n}","preventionTips":["Keep the export-flag filter in sync with the serializer's supported attribute types","When adding new attribute types, update both the filter and the arrow serializer","Treat any occurrence of this error as an upstream filtering bug and investigate the joined error list"],"tags":["arrow","serialization","telemetry","invariant"],"backgroundTag":"record-serialization-failed","analyzedSha":"0267ce9170576975b76b64ce856b2e5848e96617","analyzedAt":"2026-09-07T21:53:39.732Z","contentChangedAt":"2026-09-07T21:53:39.732Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}