pola-rs/polars · error

{:?} -> {:?} not supported

Error message

{:?} -> {:?} not supported

What it means

The final catch-all arm of `new_serializer` in crates/polars-arrow/src/io/avro/write/serialize.rs is `todo!("{:?} -> {:?} not supported", a, b)` — reached when no (Arrow physical type, Avro schema) pair matched. This is an explicit 'known gap': the combination is recognized but not implemented (unlike the unreachable! arms, which assert internal consistency). The companion predicate `can_serialize(dtype)` exists precisely to test support without panicking.

Source

Thrown at crates/polars-arrow/src/io/avro/write/serialize.rs:496

            let schema = if let AvroSchema::Array(schema) = &inner[1] {
                schema.as_ref()
            } else {
                unreachable!("The schema declaration does not match the deserialization")
            };
            list_optional::<i64>(array.as_any().downcast_ref().unwrap(), schema)
        },
        (PhysicalType::Struct, AvroSchema::Record(inner)) => {
            struct_required(array.as_any().downcast_ref().unwrap(), inner)
        },
        (PhysicalType::Struct, AvroSchema::Union(inner)) => {
            let inner = if let AvroSchema::Record(inner) = &inner[1] {
                inner
            } else {
                unreachable!("The schema declaration does not match the deserialization")
            };
            struct_optional(array.as_any().downcast_ref().unwrap(), inner)
        },
        (a, b) => todo!("{:?} -> {:?} not supported", a, b),
    }
}

/// Whether [`new_serializer`] supports `dtype`.
pub fn can_serialize(dtype: &ArrowDataType) -> bool {
    use ArrowDataType::*;
    match dtype.to_storage() {
        List(inner) => return can_serialize(&inner.dtype),
        LargeList(inner) => return can_serialize(&inner.dtype),
        Struct(inner) => return inner.iter().all(|inner| can_serialize(&inner.dtype)),
        _ => {},
    };

    matches!(
        dtype,
        Boolean
            | Int32
            | Int64

View on GitHub (pinned to 9b5d73fd00)

Solutions

1. Call `can_serialize(array.dtype().to_storage())` before writing and fail fast with a clean error.
2. Cast unsupported dtypes to supported ones first: Utf8View -> Utf8, BinaryView -> Binary, Dictionary -> dense values, FixedSizeList -> List.
3. Drop or stringify genuinely unsupported columns.
4. Check the can_serialize match list in your polars version to see exactly which dtypes are writable.

Example fix

// before
let ser = new_serializer(&utf8_view_array, &schema)?; // todo!(): views not supported

// after
use polars_arrow::io::avro::write::serialize::can_serialize;
if !can_serialize(&array.dtype().to_storage()) {
    let array = array.to_boxed().cast(&ArrowDataType::LargeUtf8)?; // view -> utf8
}
let ser = new_serializer(&array, &schema)?;

use polars_arrow::io::avro::write::serialize::can_serialize;
if !can_serialize(&array.dtype().to_storage()) {
    polars_bail!(InvalidOperation: "avro writer does not support dtype {:?}", array.dtype());
}

fn avro_writable(dtype: &ArrowDataType) -> bool {
    polars_arrow::io::avro::write::serialize::can_serialize(&dtype.to_storage())
}

let ser = std::panic::catch_unwind(|| new_serializer(&array, &schema))
    .map_err(|_| polars_err!(InvalidOperation: "arrow->avro unsupported for {:?} -> {:?}", array.dtype(), schema))?;

When it happens

Trigger: Writing an Arrow array to Avro whose dtype has no serializer: Utf8View/BinaryView strings in some paths, Dictionary-encoded arrays, fixed-size lists, null-typed columns, or any primitive whose Avro schema variant doesn't pair with the physical type (e.g. Int8 array paired with an Avro long schema).

Common situations: Polars string columns stored as Utf8View (the modern default) being handed to the Avro writer; dictionary-encoded data read from parquet then written to Avro; schema pairs where nullability or type width differs between Arrow and the Avro schema.

Related errors

• not implemented
• The schema declaration does not match the deserialization
• StructArray must be initialized with DataType::Struct
• Union struct must be created with the corresponding Union Da
• MutableUtf8ValuesArray can only be initialized with DataType

AI-assisted analysis of pola-rs/polars@9b5d73fd00 (2026-08-19). Data as JSON: /api/errors/965799750278369d. Report an issue: GitHub.