What causes "index positions should be smaller than 2^32" in pola-rs/polars?

If the values are IDs, not positions, use df.filter(pl.col('id').is_in(ids)) or join instead of indexing

pola-rs/polars · error · ValueError

index positions should be smaller than 2^32

Error message

index positions should be smaller than 2^32

What it means

Default polars wheels build with a 32-bit index dtype (UInt32). _convert_series_to_indices (getitem.py:371) rejects Int64/UInt64 Series whose maximum is >= 2**32 because such positions cannot be represented; the check only exists when get_index_type() == UInt32, so the polars-u64-idx build is unaffected.

Source

Thrown at py-polars/src/polars/_utils/getitem.py:371

    idx_type = get_index_type()

    if s.dtype == idx_type:
        return s

    if not s.dtype.is_integer():
        if s.dtype == Boolean:
            _raise_on_boolean_mask()
        else:
            msg = f"cannot treat Series of type {s.dtype} as indices"
            raise TypeError(msg)

    if s.len() == 0:
        return pl.Series(s.name, [], dtype=idx_type)

    if idx_type == UInt32:
        if s.dtype in {Int64, UInt64} and s.max() >= U32_MAX:  # type: ignore[operator]
            msg = "index positions should be smaller than 2^32"
            raise ValueError(msg)
        if s.dtype == Int64 and s.min() < -U32_MAX:  # type: ignore[operator]
            msg = "index positions should be greater than or equal to -2^32"
            raise ValueError(msg)

    if s.dtype.is_signed_integer():
        if s.min() < 0:  # type: ignore[operator]
            if idx_type == UInt32:
                idxs = s.cast(Int32) if s.dtype in {Int8, Int16} else s
            else:
                idxs = s.cast(Int64) if s.dtype in {Int8, Int16, Int32} else s

            # Update negative indexes to absolute indexes.
            return (
                idxs.to_frame()
                .select(
                    F.when(F.col(idxs.name) < 0)
                    .then(size + F.col(idxs.name))
                    .otherwise(F.col(idxs.name))

View on GitHub (pinned to df599052da)

Solutions

If the values are IDs, not positions, use df.filter(pl.col('id').is_in(ids)) or join instead of indexing
If positions are genuinely huge, install the 64-bit index wheel: pip install polars-u64-idx (it imports as polars)
Filter/collect a smaller frame first so positions fit in 32 bits

Example fix

# before
df[pl.Series(ids, dtype=pl.UInt64)]  # huge values used as positions

# after
df.filter(pl.col("id").is_in(ids))

Defensive patterns

Strategy: validation

Validate before calling

U32_MAX = 2**32
if idx.dtype in (pl.Int64, pl.UInt64) and idx.len() and idx.max() >= U32_MAX:
    raise ValueError("positions exceed 32-bit index range; use filter/join or polars-u64-idx")
df[idx]

Type guard

def fits_u32(s: "pl.Series") -> bool:
    return not (s.dtype in (pl.Int64, pl.UInt64) and s.len() and (s.max() or 0) >= 2**32)

Prevention

Keep row positions below 2**32 on default wheels
Use join/filter on an ID column when values look like large integers
Switch to polars-u64-idx for frames beyond ~4.29 billion rows

When it happens

Trigger: df[pl.Series([2**32], dtype=pl.UInt64)]; df[pl.Series([5_000_000_000], dtype=pl.Int64)]; positional gather into frames with more than ~4.29 billion rows on default wheels.

Common situations: Very large datasets on default wheels; business IDs (large integers) mistakenly used as row positions instead of values for is_in/join.

Related errors

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/ce6101e1c0cbf38b. Report an issue: GitHub.