dbt-labs/dbt-core · error · std::io::Error (InvalidData)

Failed to serialize some records: {errors}

Error message

Failed to serialize some records: {errors}

What it means

serialize_to_arrow collects per-record serialization errors into `errors`; if any remain after processing, it returns an std::io::Error with ErrorKind::InvalidData. Per the comment, this is currently treated as an internal invariant: records with non-serializable attributes should already have been filtered out upstream via export flags, so hitting this indicates either a bug in that filtering or unexpected attribute values.

Source

Thrown at crates/dbt-tracing/src/serialize/arrow.rs:617

        records
            .iter()
            .filter(|r| {
                // Only include records with serializable attributes
                r.attributes()
                    .output_flags()
                    .contains(TelemetryOutputFlags::EXPORT_PARQUET)
            })
            .filter_map(|r| {
                ArrowTelemetryRecord::try_from(r)
                    .map_err(|e| errors.push(e))
                    .ok()
            })
            .collect();

    if !errors.is_empty() {
        // As of today, this should never happen because we filter out records with non-serializable attributes
        // above via export flags and this is the only realistic error case.
        return Err(Box::new(std::io::Error::new(
            std::io::ErrorKind::InvalidData,
            format!("Failed to serialize some records: {}", errors.join("; ")),
        )));
    }

    // Serialize with the temporary schema (timestamp fields as u64),
    // see `create_arrow_schema` for details.
    let batch = serde_arrow::to_record_batch(schemas.serialisable_schema(), &arrow_records)?;

    let mut columns = batch.columns().to_vec();

    // Convert timestamp columns from u64 to Timestamp(NANOSECOND),
    // this is zero-copy, just metadata change.
    let schema_with_timestamps = schemas.schema_with_timestamps();
    for (i, field) in schema_with_timestamps.iter().enumerate() {
        if let DataType::Timestamp(TimeUnit::Nanosecond, None) = field.data_type()
            && let Some(column) = columns.get(i)
        {

View on GitHub (pinned to 0267ce9170)

Solutions

1. Inspect the joined `errors` message to find the failing record and attribute
2. Fix the export-flag filtering upstream so non-serializable records are excluded before serialize_to_arrow
3. If a new attribute type is legitimately needed, add support for it in the arrow serializer in crates/dbt-tracing/src/serialize/arrow.rs

Example fix

// before
let batch: Vec<_> = records.into_iter().collect();
// after: filter non-serializable records before serializing
let batch: Vec<_> = records
    .into_iter()
    .filter(|r| r.has_exportable_attributes())
    .collect();

let non_exportable: Vec<_> = records.iter()
    .filter(|r| !r.has_exportable_attributes())
    .collect();
if !non_exportable.is_empty() {
    // drop or fix these records before calling serialize_to_arrow
}

match serialize_to_arrow(records) {
    Err(e) if e.to_string().starts_with("Failed to serialize some records") => {
        log::error!("arrow serialization failed: {e}");
        // inspect which record/attribute failed and re-export with filters
    }
    other => other?,
}

When it happens

Trigger: Any record in the batch produces a serialization failure after the export-flag filtering pass — e.g. an attribute value the arrow serializer cannot encode slipping through the filter, or a regression in `flush_batch`'s input path.

Common situations: Telemetry/export pipelines flushing batches containing records with unexpected attribute types or values; custom record types added without updating the filter and the serializer; test-roundtrip code feeding synthetic records directly.

Related errors

• expected_field should be Struct
• get_csv_data: failed to format CSV: {e}
• invalid serialized time precision
• invalid serialized timestamp precision
• attrs_updater should only be called once

AI-assisted analysis of dbt-labs/dbt-core@0267ce9170 (2026-09-07). Data as JSON: /api/errors/110923d508d2a9f7. Report an issue: GitHub.