pola-rs/polars · error · CopyNotAllowedError

data buffer must be cast from

Error message

data buffer must be cast from {data_dtype} to UInt32

What it means

Polars stores categorical/Enum indices as UInt32. When an imported dictionary column's indices buffer has another dtype (UInt8/UInt16/Int32/Int64 — pyarrow commonly uses int8/int32 indices, pandas codes are int8..int32), Polars must cast the physical Series to UInt32, which allocates; with allow_copy=False and a non-empty column this raises CopyNotAllowedError. The Series is first built in the original dtype so out-of-range sentinels survive, then cast.

Solutions

1. Call pl.from_dataframe(obj, allow_copy=True) so the indices can be cast to UInt32.
2. Re-encode the producer's dictionary to uint32 indices before export.
3. Catch CopyNotAllowedError and retry with allow_copy=True.

Example fix

# before
df = pl.from_dataframe(obj, allow_copy=False)  # int8 indices -> CopyNotAllowedError

# after
df = pl.from_dataframe(obj, allow_copy=True)   # indices cast to UInt32

def needs_index_cast(icdf) -> bool:
    for i in range(icdf.num_columns()):
        col = icdf.get_column(i)
        if col.dtype[0] != DtypeKind.CATEGORICAL:
            continue
        if not col.describe_categorical().get("is_dictionary"):
            continue
        _, idx_dtype = col.get_buffers()["data"]
        if col.size() > 0 and (idx_dtype[0], idx_dtype[1]) != (DtypeKind.UINT, 32):
            return True
    return False

allow_copy = not needs_index_cast(obj.__dataframe__())

from polars.interchange.protocol import CopyNotAllowedError

try:
    df = pl.from_dataframe(obj, allow_copy=False)
except CopyNotAllowedError:
    df = pl.from_dataframe(obj, allow_copy=True)  # allows the UInt32 index cast

When it happens

Trigger: pl.from_dataframe(obj, allow_copy=False) where obj contains a non-empty dictionary/categorical column whose indices buffer is not uint32 — the default layout of most Arrow-based producers.

Common situations: Zero-copy import requirements colliding with Arrow's variable-width index types; dictionary columns with low cardinality (int8 indices) coming from parquet/IPC files read by pyarrow.

Related errors

• bitmask must be constructed
• bitmask must be inverted
• byte-packed boolean buffer must be converted to bit-packed…
• bytemask must be converted into a bitmask
• offsets buffer must be cast from

AI-assisted analysis of pola-rs/polars@9b5d73fd00 (2026-08-19). Data as JSON: /api/errors/6739b7d17c2b1708. Report an issue: GitHub.

Appendix: source

Thrown at py-polars/src/polars/interchange/from_dataframe.py:183

    data_buffer = _construct_data_buffer(
        *buffers["data"], column.size(), offset, allow_copy=allow_copy
    )
    validity_buffer = _construct_validity_buffer(
        buffers["validity"], column, dtype, data_buffer, offset, allow_copy=allow_copy
    )

    # First construct a physical Series without categories
    # to allow for sentinel values that do not fit in UInt32
    data_dtype = data_buffer.dtype
    out = pl.Series._from_buffers(
        data_dtype, data=data_buffer, validity=validity_buffer
    )

    # Polars only supports UInt32 categoricals
    if data_dtype != UInt32:
        if not allow_copy and column.size() > 0:
            msg = f"data buffer must be cast from {data_dtype} to UInt32"
            raise CopyNotAllowedError(msg)

        # TODO: Cast directly to Enum
        # https://github.com/pola-rs/polars/issues/13409
        out = out.cast(UInt32)

    return out.cast(dtype)


def _construct_data_buffer(
    buffer: Buffer,
    dtype: Dtype,
    length: int,
    offset: int = 0,
    *,
    allow_copy: bool,
) -> Series:
    polars_dtype = dtype_to_polars_dtype(dtype)

View on GitHub (pinned to 9b5d73fd00)