pola-rs/polars · error

invalid value for POLARS_MAX_OPEN_SINKS

Error message

invalid value for POLARS_MAX_OPEN_SINKS: {x}

What it means

Polars reads POLARS_MAX_OPEN_SINKS to cap how many sink files may be open concurrently in a partitioned sink. An unparsable (or zero) value panics at config resolution time, since a non-zero usize is required.

Solutions

  1. Set a positive integer, e.g. export POLARS_MAX_OPEN_SINKS=32.
  2. Unset the variable to fall back to defaults (512 for cloud targets, otherwise the internal default).
  3. Remove any 'unlimited' style values; there is no sentinel — only a positive number or unset.

Example fix

# before
POLARS_MAX_OPEN_SINKS=0
# after
POLARS_MAX_OPEN_SINKS=32
Defensive patterns

Strategy: validation

Validate before calling

import os
v = os.environ.get("POLARS_MAX_OPEN_SINKS")
if v is not None:
    assert v.isdigit() and int(v) > 0, f"POLARS_MAX_OPEN_SINKS must be a positive integer, got {v!r}"

Type guard

def is_positive_int(s: str) -> bool:
    return s.isdigit() and int(s) > 0

Try / catch

if not (os.environ.get("POLARS_MAX_OPEN_SINKS") or "1").isdigit() or int(os.environ["POLARS_MAX_OPEN_SINKS"]) < 1:
    del os.environ["POLARS_MAX_OPEN_SINKS"]  # fall back to defaults

Prevention

When it happens

Trigger: Setting POLARS_MAX_OPEN_SINKS to a non-positive-integer value ('0', 'ten', '', '2.5') and then running a partitioned sink (sink_parquet with partition_by, which calls max_open_sinks from start_partition_sink_pipeline).

Common situations: Tuning concurrent file writes for partitioned output and mistyping the value; container/k8s manifests with malformed env entries; using '0' intending 'unlimited'.

Understand the failure class

Background: "is not a valid" / "Invalid ... value" environment variable errors: how libraries validate env vars and what to do when they reject yours — this error's family across 48 libraries.

Related errors

AI-assisted analysis of pola-rs/polars@fe841f959e (2026-09-18). Data as JSON: /api/errors/2952dcbfc542dff6. Report an issue: GitHub.

Appendix: source

Thrown at crates/polars-stream/src/nodes/io_sinks/config.rs:49

        if let Ok(v) = std::env::var("POLARS_INFLIGHT_SINK_MORSEL_LIMIT").map(|x| {
            x.parse::<NonZeroUsize>().unwrap_or_else(|_| {
                panic!("invalid value for POLARS_INFLIGHT_SINK_MORSEL_LIMIT: {x}")
            })
        }) {
            return v;
        };

        NonZeroUsize::saturating_add(
            num_pipelines,
            // Additional buffer to accommodate head-of-line blocking
            4,
        )
    }

    pub fn max_open_sinks(&self) -> NonZeroUsize {
        if let Ok(v) = std::env::var("POLARS_MAX_OPEN_SINKS").map(|x| {
            x.parse::<NonZeroUsize>()
                .unwrap_or_else(|_| panic!("invalid value for POLARS_MAX_OPEN_SINKS: {x}"))
        }) {
            return v;
        }

        if self.target.is_cloud_location() {
            const { NonZeroUsize::new(512).unwrap() }
        } else {
            const { NonZeroUsize::new(128).unwrap() }
        }
    }

    pub fn cloud_upload_chunk_size(&self) -> Option<NonZeroUsize> {
        // Only when `is_cloud_location() == true`. This excludes FORCE_ASYNC'ed local paths,
        // to avoid unnecessary memory copying.
        polars_io::configs::env_upload_chunk_size().or((self.target.is_cloud_location()
            || self.file_write_format_requires_buffering())
        .then_some(DEFAULT_UPLOAD_CHUNK_SIZE))
    }

View on GitHub (pinned to fe841f959e)