pola-rs/polars · error

The schema declaration does not match the deserialization

Error message

The schema declaration does not match the deserialization

What it means

In crates/polars-arrow/src/io/avro/write/serialize.rs, `new_serializer(array, avro_schema)` pairs the array's physical type with an Avro schema you supply. For `(PhysicalType::List, AvroSchema::Union(inner))` it assumes the union is exactly [null, Array] and indexes `inner[1]`; if that slot is not an `AvroSchema::Array`, it panics via `unreachable!` with 'The schema declaration does not match the deserialization'. The crate assumes array and schema were produced together, so any divergence is considered impossible.

Source

Thrown at crates/polars-arrow/src/io/avro/write/serialize.rs:473

                    } else {
                        buf.push(IS_NULL);
                    }
                },
                vec![],
            ))
        },

        (PhysicalType::List, AvroSchema::Array(schema)) => {
            list_required::<i32>(array.as_any().downcast_ref().unwrap(), schema.as_ref())
        },
        (PhysicalType::LargeList, AvroSchema::Array(schema)) => {
            list_required::<i64>(array.as_any().downcast_ref().unwrap(), schema.as_ref())
        },
        (PhysicalType::List, AvroSchema::Union(inner)) => {
            let schema = if let AvroSchema::Array(schema) = &inner[1] {
                schema.as_ref()
            } else {
                unreachable!("The schema declaration does not match the deserialization")
            };
            list_optional::<i32>(array.as_any().downcast_ref().unwrap(), schema)
        },
        (PhysicalType::LargeList, AvroSchema::Union(inner)) => {
            let schema = if let AvroSchema::Array(schema) = &inner[1] {
                schema.as_ref()
            } else {
                unreachable!("The schema declaration does not match the deserialization")
            };
            list_optional::<i64>(array.as_any().downcast_ref().unwrap(), schema)
        },
        (PhysicalType::Struct, AvroSchema::Record(inner)) => {
            struct_required(array.as_any().downcast_ref().unwrap(), inner)
        },
        (PhysicalType::Struct, AvroSchema::Union(inner)) => {
            let inner = if let AvroSchema::Record(inner) = &inner[1] {
                inner
            } else {

View on GitHub (pinned to 9b5d73fd00)

Solutions

1. Derive the Avro schema from the Arrow dtype instead of passing an independent one, so the union layout always matches ([null, Array] for nullable lists).
2. Pre-check the union: require `matches!(inner.as_slice(), [AvroSchema::Null, AvroSchema::Array(_)])` before serializing.
3. If the schema is authoritative, convert the Arrow array to match it (cast/cast nullability) before calling new_serializer.
4. Wrap the write in catch_unwind at the API boundary to return an error naming the mismatched field.

Example fix

// before
let ser = new_serializer(&list_array, &hand_written_avro_schema)?; // panics if union[1] != Array

// after
fn union_ok(dtype: &ArrowDataType, schema: &AvroSchema) -> bool {
    use polars_arrow::datatypes::PhysicalType;
    match (dtype.to_physical_type(), schema) {
        (PhysicalType::List, AvroSchema::Union(inner))
        | (PhysicalType::LargeList, AvroSchema::Union(inner)) =>
            matches!(inner.as_slice(), [AvroSchema::Null, AvroSchema::Array(_)]),
        (PhysicalType::Struct, AvroSchema::Union(inner)) =>
            matches!(inner.as_slice(), [AvroSchema::Null, AvroSchema::Record(_)]),
        _ => true,
    }
}
assert!(union_ok(list_array.dtype(), &schema), "avro union does not match arrow dtype");

fn avro_union_matches(dtype: &ArrowDataType, schema: &AvroSchema) -> bool {
    use polars_arrow::datatypes::PhysicalType;
    match (dtype.to_physical_type(), schema) {
        (PhysicalType::List, AvroSchema::Union(inner)) =>
            matches!(inner.as_slice(), [AvroSchema::Null, AvroSchema::Array(_)]),
        _ => true,
    }
}
assert!(avro_union_matches(array.dtype(), &schema), "avro union does not match List dtype");

fn is_polars_nullable_list_union(schema: &AvroSchema) -> bool {
    matches!(schema, AvroSchema::Union(inner) if matches!(inner.as_slice(), [AvroSchema::Null, AvroSchema::Array(_)]))
}

let ser = std::panic::catch_unwind(|| new_serializer(&array, &schema))
    .map_err(|_| polars_err!(InvalidOperation: "avro schema does not match arrow dtype {:?}", array.dtype()))?;

When it happens

Trigger: Calling `new_serializer` (or the Avro writer that drives it) with a nullable List array but an Avro schema whose union's second variant is not Array — e.g. [null, null], [String, Array], or a union with a different member order. This happens when the Avro schema is hand-written, parsed from a .avsc file, or comes from another tool rather than being derived from the Arrow dtype.

Common situations: Writing Arrow data to Avro with an externally supplied writer schema; schema evolution where the .avsc union was edited; mixing schema sources between write and read paths. The sibling arms at :481 (LargeList) and :492 (Struct) fail identically.

Related errors

• {:?} -> {:?} not supported
• Union struct must be created with the corresponding Union Da
• cannot set validity of a union array
• not implemented
• length to fit in `usize`

AI-assisted analysis of pola-rs/polars@9b5d73fd00 (2026-08-19). Data as JSON: /api/errors/7048a9ecbd03e5fb. Report an issue: GitHub.