Polars' maximum length reached. Consider compiling with 'big

Error message

Polars' maximum length reached. Consider compiling with 'bigidx' feature.

What it means

By default polars indexes rows with IdxSize = u32 (max ~4.29 billion). NullChunked::new (used for any Null-dtyped series creation) checks the requested length and panics with LENGTH_LIMIT_MSG when it reaches IdxSize::MAX, advising to compile with the bigidx feature which switches IdxSize to u64.

Source

Thrown at crates/polars-core/src/series/implementations/null.rs:30

impl Series {
    pub fn new_null(name: PlSmallStr, len: usize) -> Series {
        NullChunked::new(name, len).into_series()
    }
}

#[derive(Clone)]
pub struct NullChunked {
    pub(crate) name: PlSmallStr,
    length: usize,
    // we still need chunks as many series consumers expect
    // chunks to be there
    chunks: Vec<ArrayRef>,
}

impl NullChunked {
    pub(crate) fn new(name: PlSmallStr, len: usize) -> Self {
        if len >= (IdxSize::MAX as usize) && chunkops::CHECK_LENGTH.get() {
            panic!("{}", LENGTH_LIMIT_MSG);
        }

        Self {
            name,
            length: len,
            chunks: vec![Box::new(arrow::array::NullArray::new(
                ArrowDataType::Null,
                len,
            ))],
        }
    }

    pub fn len(&self) -> usize {
        self.length
    }

    pub fn is_empty(&self) -> bool {
        self.length == 0

View on GitHub (pinned to 68506541d2)

Solutions

Enable the bigidx feature (Rust: features = ["bigidx"]) so indexes become u64
Reduce the materialized length: process in chunks, or keep the operation lazy/streaming so rows are never fully materialized
Guard user-supplied sizes against the 2^32 limit and fail with a clear error before calling into polars

Example fix

# before
s = pl.repeat(None, 5_000_000_000, dtype=pl.Null, eager=True)  # panics

# after
# rust: polars = { features = ["bigidx"] }
# or keep it lazy / chunk the work
lf = pl.select(pl.repeat(None, 5_000_000_000, dtype=pl.Null)).lazy()

Defensive patterns

Strategy: validation

Validate before calling

const IDX_MAX: usize = u32::MAX as usize; // default IdxSize
fn length_allowed(len: usize) -> bool {
    len < IDX_MAX
}

Prevention

Enable the bigidx feature when data can exceed ~4.29 billion rows
Guard user-supplied sizes/lengths against 2^32 before materializing Series
Keep huge operations lazy/streaming so rows are never fully materialized in one Series

When it happens

Trigger: Materializing a Null/empty-typed Series with length >= 2^32: pl.Series of dtype Null with a huge size hint, range(5_000_000_000) eager materialization, giant broadcasts, or cross joins that create more than 4.29e9 rows on a default (non-bigidx) build.

Common situations: Large-scale ETL (multi-billion row ranges, cross joins, reindexes) on the standard wheel/build; code ported from systems with 64-bit indexes; tests with synthetic huge lengths.

Related errors

AI-assisted analysis of pola-rs/polars@68506541d2 (2026-08-19). Data as JSON: /api/errors/508e09dcbee54893. Report an issue: GitHub.