pola-rs/polars · error

cannot set validity of a union array

Error message

cannot set validity of a union array

What it means

UnionArray's Array impl hard-codes with_validity to panic and validity() to return None: the Arrow union layout has no top-level validity bitmap, so attaching one is rejected rather than silently dropped. Nullability lives in the child fields.

Source

Thrown at crates/polars-arrow/src/array/union/mod.rs:349

    pub unsafe fn value_unchecked(&self, index: usize) -> Box<dyn Scalar> {
        debug_assert!(index < self.len());
        let (type_, index) = self.index_unchecked(index);
        // SAFETY: assumption of the struct
        debug_assert!(type_ < self.fields.len());
        let field = self.fields.get_unchecked(type_).as_ref();
        new_scalar(field, index)
    }
}

impl Array for UnionArray {
    impl_common_array!();

    fn validity(&self) -> Option<&Bitmap> {
        None
    }

    fn with_validity(&self, _: Option<Bitmap>) -> Box<dyn Array> {
        panic!("cannot set validity of a union array")
    }
}

impl UnionArray {
    fn try_get_all(dtype: &ArrowDataType) -> PolarsResult<UnionComponents<'_>> {
        match dtype.to_storage() {
            ArrowDataType::Union(u) => Ok((&u.fields, u.ids.as_ref().map(|x| x.as_ref()), u.mode)),
            _ => polars_bail!(ComputeError:
                "The UnionArray requires a logical type of DataType::Union",
            ),
        }
    }

    fn get_all(dtype: &ArrowDataType) -> (&[Field], Option<&[i32]>, UnionMode) {
        Self::try_get_all(dtype).unwrap()
    }

    /// Returns all fields from [`ArrowDataType::Union`].

View on GitHub (pinned to 9b5d73fd00)

Solutions

1. Branch first: if matches!(arr.dtype().to_physical_type(), PhysicalType::Union), skip validity handling for that array
2. Model top-level nullability by wrapping the union in a StructArray that carries the validity, or make the relevant union child field nullable
3. In generic pipelines, apply validity to child arrays (zip_validity-style) instead of blind with_validity on the top level

Example fix

// before
let out = arr.with_validity(Some(validity));

// after
let out = if matches!(arr.dtype().to_physical_type(), PhysicalType::Union) {
    arr.to_boxed() // unions cannot carry top-level validity
} else {
    arr.with_validity(Some(validity))
};

fn can_set_validity(arr: &dyn polars_arrow::array::Array) -> bool {
    !matches!(arr.dtype().to_physical_type(), polars_arrow::datatypes::PhysicalType::Union)
}

fn is_union_array(arr: &dyn polars_arrow::array::Array) -> bool {
    matches!(arr.dtype().to_physical_type(), polars_arrow::datatypes::PhysicalType::Union)
}

When it happens

Trigger: Generic code doing arr.with_validity(Some(bitmap)) on Box<dyn Array>/&dyn Array when the concrete array is a UnionArray — typical in filter/zip/concat-style kernels that recompute a validity and re-apply it uniformly to the output.

Common situations: Reusing a kernel across a heterogeneous schema that includes a union column; porting code from arrays that do support validity; wrapping foreign arrow union data into polars pipelines.

Related errors

• Union struct must be created with the corresponding Union Da
• The schema declaration does not match the deserialization
• activate 'dtype-array'
• StructArray must be initialized with DataType::Struct
• The length of the values must be equal to the last offset va

AI-assisted analysis of pola-rs/polars@9b5d73fd00 (2026-08-19). Data as JSON: /api/errors/ac5b23bc6116749f. Report an issue: GitHub.