pola-rs/polars · error

invalid value for POLARS_NDJSON_CHUNK_SIZE: {x}

Error message

invalid value for POLARS_NDJSON_CHUNK_SIZE: {x}

What it means

Environment-variable parsing guard in the streaming NDJSON source: the POLARS_NDJSON_CHUNK_SIZE override was set to a value that cannot be parsed as the expected integer type. The env var's raw value is the faulty input.

Source

Thrown at crates/polars-stream/src/nodes/io_sources/ndjson/mod.rs:430

        let row_skipper = RowSkipper {
            cfg_n_rows_to_skip: n_rows_to_skip,
            n_rows_skipped: 0,
            is_line: self.chunk_reader_builder.is_line_fn(),
            reverse: is_negative_slice,
        };

        // Unify the two source options (uncompressed local file mmapp'ed, or streaming async with
        // transparent decompression), into one unified reader source.
        let reader_source = if use_async_prefetch {
            // Prepare parameters for Prefetch task.
            // TODO REFACTOR: use get_streaming_chunk_size()
            const DEFAULT_NDJSON_CHUNK_SIZE: usize = 32 * 1024 * 1024;
            let memory_prefetch_func = get_memory_prefetch_func(verbose);
            let chunk_size = std::env::var("POLARS_NDJSON_CHUNK_SIZE")
                .map(|x| {
                    x.parse::<NonZeroUsize>()
                        .unwrap_or_else(|_| {
                            panic!("invalid value for POLARS_NDJSON_CHUNK_SIZE: {x}")
                        })
                        .get()
                })
                .unwrap_or(DEFAULT_NDJSON_CHUNK_SIZE);

            let prefetch_limit = self
                .chunk_prefetch_sync
                .prefetch_limit
                .min(file_size.div_ceil(chunk_size))
                .max(1);

            let (prefetch_send, prefetch_recv) = tokio::sync::mpsc::channel(prefetch_limit);

            // Task: Prefetch.
            // Initiate parallel downloads of raw data chunks.
            let byte_source = byte_source.clone();
            let prefetch_task = {
                let prefetch_semaphore = Arc::clone(&self.chunk_prefetch_sync.prefetch_semaphore);

View on GitHub (pinned to 5d8ebabf11)

Solutions

1. Set POLARS_NDJSON_CHUNK_SIZE to a positive integer or unset it.

Example fix

export POLARS_NDJSON_CHUNK_SIZE=8192

When it happens

Trigger: Invalid value in POLARS_NDJSON_CHUNK_SIZE.

Common situations: See trigger scenarios.

AI-assisted analysis of pola-rs/polars@5d8ebabf11 (2026-08-19). Data as JSON: /api/errors/3133c2f66de2d902. Report an issue: GitHub.