pola-rs/polars · error

invalid value for POLARS_ROW_GROUP_PREFETCH_SIZE: {x}

Error message

invalid value for POLARS_ROW_GROUP_PREFETCH_SIZE: {x}

What it means

Panic in ParquetReaderBuilder::set_execution_state: POLARS_ROW_GROUP_PREFETCH_SIZE was set but is not a valid NonZeroUsize. The env var overrides the number of row groups prefetched into the decode pipeline (defaulting to a clamp of pipelines-based heuristic), so the bad value {x} aborts reader setup.

Source

Thrown at crates/polars-stream/src/nodes/io_sources/parquet/builder.rs:76

        if matches!(
            self.options.parallel,
            ParallelStrategy::Auto | ParallelStrategy::Prefiltered
        ) {
            capabilities |= RC::FULL_FILTER;
        }
        capabilities
    }

    fn set_execution_state(&self, execution_state: &crate::execute::StreamingExecutionState) {
        // Bound the number of fetches in the pipeline.
        // This bound goes together with the `prefetch_kbytes_limit` bound. In most
        // large-dataset use cases, the kbytes memory bound will kick in first.
        // This limit should be at least as large as the max in-flight concurrency.
        let prefetch_limit = std::env::var("POLARS_ROW_GROUP_PREFETCH_SIZE")
            .map(|x| {
                x.parse::<NonZeroUsize>()
                    .unwrap_or_else(|_| {
                        panic!("invalid value for POLARS_ROW_GROUP_PREFETCH_SIZE: {x}")
                    })
                    .get()
            })
            .unwrap_or(
                execution_state
                    .num_pipelines
                    .saturating_mul(2)
                    .max(get_request_budget() as usize)
                    .clamp(16, 2048),
            )
            .max(1);

        // Bound the max memory in use for the pipeline.
        // This should be large enough to be non-blocking, but small enough to avoid
        // excessive memory use from a run-away prefetch pipeline.
        // "Correct" formula: (a) max effective in-flight bdp-based bytes budget + (b) decode pipeline.
        // Since we do not know (a) at startup, we use  '3 * (b) + buffer' as a proxy for (a), where the
        // multiplier reflects the max gain factor for in-flight control.

View on GitHub (pinned to 68506541d2)

Solutions

1. Set POLARS_ROW_GROUP_PREFETCH_SIZE to a positive integer or unset it.

Example fix

export POLARS_ROW_GROUP_PREFETCH_SIZE=8

When it happens

Trigger: Invalid value in POLARS_ROW_GROUP_PREFETCH_SIZE.

Common situations: See trigger scenarios.

AI-assisted analysis of pola-rs/polars@68506541d2 (2026-08-19). Data as JSON: /api/errors/c4b750d7c318fb57. Report an issue: GitHub.