pola-rs/polars · critical

Polars' maximum length reached. Consider compiling with…

Error message

Polars' maximum length reached. Consider compiling with 'bigidx' feature.

What it means

IdxSize (the default index type) is 32-bit, so a ChunkedArray may hold at most IdxSize::MAX - 1 (~4.29 billion) elements; IdxSize::MAX is reserved to mark NULL indices. compute_len panics once the computed chunked length reaches that limit, telling the user to recompile with the bigidx feature (64-bit indices).

Solutions

1. Switch to the bigidx feature: add `features = ["bigidx"]` to polars in Cargo.toml (the prebuilt polars-lts-cpu/polars wheels are already bigidx)
2. Split the work into partitions smaller than u32::MAX rows and process them separately
3. Reduce data by filtering/projecting before materializing the full column

Example fix

// before
polars = "0.x"
// after
polars = { version = "0.x", features = ["bigidx"] }

const IDX_MAX: usize = u32::MAX as usize - 1;
if total_rows >= IDX_MAX {
    return Err("length exceeds 32-bit IdxSize; enable bigidx or partition data");
}

When it happens

Trigger: Concatenating/extend/applying operations on chunks whose combined length reaches or exceeds u32::MAX (~4,294,967,295) rows while polars is compiled without the bigidx feature.

Common situations: Processing very large datasets (e.g. massive parquet/csv concatenations) with the default 32-bit build; hitting the limit after many concat/extend calls rather than in one operation.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• {0}
• activate dtype
• activate dtype-categorical to convert dictionary arrays
• activate ' ' feature
• activate feature dtype-date

AI-assisted analysis of pola-rs/polars@fe841f959e (2026-09-18). Data as JSON: /api/errors/b34d3e080346fd2a. Report an issue: GitHub.

Appendix: source

Thrown at crates/polars-core/src/chunked_array/ops/chunkops.rs:153

    /// Check if ChunkedArray is empty.
    #[inline]
    pub fn is_empty(&self) -> bool {
        self.len() == 0
    }

    /// Compute the length
    pub(crate) fn compute_len(&mut self) {
        fn inner(chunks: &[ArrayRef]) -> usize {
            match chunks.len() {
                // fast path
                1 => chunks[0].len(),
                _ => chunks.iter().fold(0, |acc, arr| acc + arr.len()),
            }
        }
        let len = inner(&self.chunks);
        // Length limit is `IdxSize::MAX - 1`. We use `IdxSize::MAX` to indicate `NULL` in indexing.
        if len >= (IdxSize::MAX as usize) && CHECK_LENGTH.get() {
            panic!("{}", LENGTH_LIMIT_MSG);
        }
        self.length = len;
        self.null_count = self
            .chunks
            .iter()
            .map(|arr| arr.null_count())
            .sum::<usize>();
    }

    /// Rechunks this ChunkedArray, returning a new Cow::Owned ChunkedArray if it was
    /// rechunked or simply a Cow::Borrowed of itself if it was already a single chunk.
    pub fn rechunk(&self) -> Cow<'_, Self> {
        match self.dtype() {
            #[cfg(feature = "object")]
            DataType::Object(_) => {
                panic!("implementation error")
            },
            _ => {

View on GitHub (pinned to fe841f959e)