pola-rs/polars · error · CopyNotAllowedError

data buffer must be cast from {data_dtype} to UInt32

Error message

data buffer must be cast from {data_dtype} to UInt32

What it means

Polars stores categorical dictionary indices physically as UInt32. When a producer supplies indices in another width (Int8/Int16/Int32/Int64), Polars first builds the physical Series in the original dtype - so sentinel values that do not fit UInt32 survive - and then casts to UInt32 before casting to the Enum dtype. The cast allocates, so with allow_copy=False and a non-empty column it raises CopyNotAllowedError.

Source

Thrown at py-polars/src/polars/interchange/from_dataframe.py:183

    data_buffer = _construct_data_buffer(
        *buffers["data"], column.size(), offset, allow_copy=allow_copy
    )
    validity_buffer = _construct_validity_buffer(
        buffers["validity"], column, dtype, data_buffer, offset, allow_copy=allow_copy
    )

    # First construct a physical Series without categories
    # to allow for sentinel values that do not fit in UInt32
    data_dtype = data_buffer.dtype
    out = pl.Series._from_buffers(
        data_dtype, data=data_buffer, validity=validity_buffer
    )

    # Polars only supports UInt32 categoricals
    if data_dtype != UInt32:
        if not allow_copy and column.size() > 0:
            msg = f"data buffer must be cast from {data_dtype} to UInt32"
            raise CopyNotAllowedError(msg)

        # TODO: Cast directly to Enum
        # https://github.com/pola-rs/polars/issues/13409
        out = out.cast(UInt32)

    return out.cast(dtype)


def _construct_data_buffer(
    buffer: Buffer,
    dtype: Dtype,
    length: int,
    offset: int = 0,
    *,
    allow_copy: bool,
) -> Series:
    polars_dtype = dtype_to_polars_dtype(dtype)

View on GitHub (pinned to df599052da)

Solutions

1. Use allow_copy=True for data containing dictionary-encoded categoricals
2. Have the producer emit UInt32 indices buffers
3. Catch CopyNotAllowedError and retry with allow_copy=True at the call site

Example fix

// before
df = pl.from_dataframe(producer_df, allow_copy=False)  # indices are Int32 -> error

// after
try:
    df = pl.from_dataframe(producer_df, allow_copy=False)
except pl.exceptions.CopyNotAllowedError:
    df = pl.from_dataframe(producer_df, allow_copy=True)

def categorical_indices_are_uint32(df) -> bool:
    from polars.interchange.protocol import DtypeKind
    proto = df.__dataframe__(allow_copy=False)
    for col in proto.get_columns():
        if col.dtype[0] == DtypeKind.CATEGORICAL and col.size() > 0:
            _, idx_dtype = col.get_buffers()['data']
            if idx_dtype[1] != 32 or idx_dtype[2] != 'I':
                return False
    return True

try:
    out = pl.from_dataframe(df, allow_copy=False)
except pl.exceptions.CopyNotAllowedError:
    # UInt32 cast for dictionary indices is unavoidable; allow the copy
    out = pl.from_dataframe(df, allow_copy=True)

When it happens

Trigger: pl.from_dataframe(df, allow_copy=False) on a dictionary-encoded categorical column whose indices buffer dtype is not UInt32.

Common situations: Zero-copy ingest paths receiving Int32 or Int64 dictionary indices from Arrow-based or custom producers; producers that also use out-of-range sentinel values, forcing the intermediate physical Series.

Related errors

• non-contiguous buffer must be made contiguous
• unevenly chunked columns must be rechunked
• string buffers must be converted
• byte-packed boolean buffer must be converted to bit-packed b
• offsets buffer must be cast from {polars_dtype} to Int64

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/6739b7d17c2b1708. Report an issue: GitHub.