{"record":{"id":"7048a9ecbd03e5fb","repo":"pola-rs/polars","slug":"the-schema-declaration-does-not-match-the-deserial","errorCode":null,"errorMessage":"The schema declaration does not match the deserialization","messagePattern":"The schema declaration does not match the deserialization","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/polars-arrow/src/io/avro/write/serialize.rs","lineNumber":473,"sourceCode":"                    } else {\n                        buf.push(IS_NULL);\n                    }\n                },\n                vec![],\n            ))\n        },\n\n        (PhysicalType::List, AvroSchema::Array(schema)) => {\n            list_required::<i32>(array.as_any().downcast_ref().unwrap(), schema.as_ref())\n        },\n        (PhysicalType::LargeList, AvroSchema::Array(schema)) => {\n            list_required::<i64>(array.as_any().downcast_ref().unwrap(), schema.as_ref())\n        },\n        (PhysicalType::List, AvroSchema::Union(inner)) => {\n            let schema = if let AvroSchema::Array(schema) = &inner[1] {\n                schema.as_ref()\n            } else {\n                unreachable!(\"The schema declaration does not match the deserialization\")\n            };\n            list_optional::<i32>(array.as_any().downcast_ref().unwrap(), schema)\n        },\n        (PhysicalType::LargeList, AvroSchema::Union(inner)) => {\n            let schema = if let AvroSchema::Array(schema) = &inner[1] {\n                schema.as_ref()\n            } else {\n                unreachable!(\"The schema declaration does not match the deserialization\")\n            };\n            list_optional::<i64>(array.as_any().downcast_ref().unwrap(), schema)\n        },\n        (PhysicalType::Struct, AvroSchema::Record(inner)) => {\n            struct_required(array.as_any().downcast_ref().unwrap(), inner)\n        },\n        (PhysicalType::Struct, AvroSchema::Union(inner)) => {\n            let inner = if let AvroSchema::Record(inner) = &inner[1] {\n                inner\n            } else {","sourceCodeStart":455,"sourceCodeEnd":491,"githubUrl":"https://github.com/pola-rs/polars/blob/9b5d73fd00236295624374b075d16b1fe6ec6df9/crates/polars-arrow/src/io/avro/write/serialize.rs#L455-L491","documentation":"In crates/polars-arrow/src/io/avro/write/serialize.rs, `new_serializer(array, avro_schema)` pairs the array's physical type with an Avro schema you supply. For `(PhysicalType::List, AvroSchema::Union(inner))` it assumes the union is exactly [null, Array] and indexes `inner[1]`; if that slot is not an `AvroSchema::Array`, it panics via `unreachable!` with 'The schema declaration does not match the deserialization'. The crate assumes array and schema were produced together, so any divergence is considered impossible.","triggerScenarios":"Calling `new_serializer` (or the Avro writer that drives it) with a nullable List array but an Avro schema whose union's second variant is not Array — e.g. [null, null], [String, Array], or a union with a different member order. This happens when the Avro schema is hand-written, parsed from a .avsc file, or comes from another tool rather than being derived from the Arrow dtype.","commonSituations":"Writing Arrow data to Avro with an externally supplied writer schema; schema evolution where the .avsc union was edited; mixing schema sources between write and read paths. The sibling arms at :481 (LargeList) and :492 (Struct) fail identically.","solutions":["Derive the Avro schema from the Arrow dtype instead of passing an independent one, so the union layout always matches ([null, Array] for nullable lists).","Pre-check the union: require `matches!(inner.as_slice(), [AvroSchema::Null, AvroSchema::Array(_)])` before serializing.","If the schema is authoritative, convert the Arrow array to match it (cast/cast nullability) before calling new_serializer.","Wrap the write in catch_unwind at the API boundary to return an error naming the mismatched field."],"exampleFix":"// before\nlet ser = new_serializer(&list_array, &hand_written_avro_schema)?; // panics if union[1] != Array\n\n// after\nfn union_ok(dtype: &ArrowDataType, schema: &AvroSchema) -> bool {\n    use polars_arrow::datatypes::PhysicalType;\n    match (dtype.to_physical_type(), schema) {\n        (PhysicalType::List, AvroSchema::Union(inner))\n        | (PhysicalType::LargeList, AvroSchema::Union(inner)) =>\n            matches!(inner.as_slice(), [AvroSchema::Null, AvroSchema::Array(_)]),\n        (PhysicalType::Struct, AvroSchema::Union(inner)) =>\n            matches!(inner.as_slice(), [AvroSchema::Null, AvroSchema::Record(_)]),\n        _ => true,\n    }\n}\nassert!(union_ok(list_array.dtype(), &schema), \"avro union does not match arrow dtype\");","handlingStrategy":"validation","validationCode":"fn avro_union_matches(dtype: &ArrowDataType, schema: &AvroSchema) -> bool {\n    use polars_arrow::datatypes::PhysicalType;\n    match (dtype.to_physical_type(), schema) {\n        (PhysicalType::List, AvroSchema::Union(inner)) =>\n            matches!(inner.as_slice(), [AvroSchema::Null, AvroSchema::Array(_)]),\n        _ => true,\n    }\n}\nassert!(avro_union_matches(array.dtype(), &schema), \"avro union does not match List dtype\");","typeGuard":"fn is_polars_nullable_list_union(schema: &AvroSchema) -> bool {\n    matches!(schema, AvroSchema::Union(inner) if matches!(inner.as_slice(), [AvroSchema::Null, AvroSchema::Array(_)]))\n}","tryCatchPattern":"let ser = std::panic::catch_unwind(|| new_serializer(&array, &schema))\n    .map_err(|_| polars_err!(InvalidOperation: \"avro schema does not match arrow dtype {:?}\", array.dtype()))?;","preventionTips":["Always derive the Avro writer schema from the Arrow dtype in the same process.","Canonicalize nullable unions to [null, T] before serializing.","Never feed hand-edited .avsc schemas directly to new_serializer."],"tags":["rust","polars","arrow","avro","schema-mismatch","list","union","panic"],"backgroundTag":"avro-schema-mismatch","analyzedSha":"9b5d73fd00236295624374b075d16b1fe6ec6df9","analyzedAt":"2026-08-19T12:15:06.350Z","contentChangedAt":"2026-08-19T12:15:06.350Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}