pola-rs/polars · error

invalid value for POLARS_IO_SINK_BYTES_BUFFERER_CONFIG

Error message

invalid value for POLARS_IO_SINK_BYTES_BUFFERER_CONFIG: {v}

What it means

POLARS_IO_SINK_BYTES_BUFFERER_CONFIG is an optional env var carrying a structured size config (target size range and copy-buffer reserve) for sink byte buffering. Polars parses it into a validated struct; if the string is present but malformed, inconsistent (end < start), has trailing tokens, or reserves are inconsistent, it panics.

Solutions

1. Correct the value so both ranges satisfy start <= end and the whole string is consumed by the parser.
2. Unset POLARS_IO_SINK_BYTES_BUFFERER_CONFIG to use built-in defaults.
3. If tuning upload chunking, adjust POLARS_IO_SINK cloud upload chunk settings instead of hand-crafting this config.

Example fix

# before (truncated / invalid)
export POLARS_IO_SINK_BYTES_BUFFERER_CONFIG="(1..0, 2..3"
# after
unset POLARS_IO_SINK_BYTES_BUFFERER_CONFIG

import os
v = os.environ.get("POLARS_IO_SINK_BYTES_BUFFERER_CONFIG")
if v is not None:
    # must be fully consumable with ordered ranges; if unsure of format, unset
    print(f"warning: custom bufferer config set: {v!r}")

def bufferer_config_suspect(s: str) -> bool:
    return s.strip() == "" or ".." not in s

# Cannot catch the panic; defensively unset questionable values before importing/running polars sink:
v = os.environ.get("POLARS_IO_SINK_BYTES_BUFFERER_CONFIG")
if v and (".." not in v):
    del os.environ["POLARS_IO_SINK_BYTES_BUFFERER_CONFIG"]

When it happens

Trigger: Setting POLARS_IO_SINK_BYTES_BUFFERER_CONFIG to a malformed tuple/string and starting any file sink (single file or partitioned) via bytes_bufferer_config; the parse must yield valid ordering (end >= start for both ranges) and consume the entire value.

Common situations: Copy-pasting an internal debug config string with wrong separators or order-swapped ranges; partial truncation of a long env value; guessing the format instead of the documented one.

Understand the failure class

Background: "is not a valid" / "Invalid ... value" environment variable errors: how libraries validate env vars and what to do when they reject yours — this error's family across 48 libraries.

Related errors

• Invalid `POLARS_PQ_PREFILTERED_MASK` value
• invalid value for POLARS_CSV_CHUNK_PREFETCH_LIMIT
• invalid value for POLARS_IDEAL_SINK_MORSEL_SIZE_BYTES
• invalid value for POLARS_IDEAL_SINK_MORSEL_SIZE_ROWS
• invalid value for POLARS_INFLIGHT_SINK_MORSEL_LIMIT

AI-assisted analysis of pola-rs/polars@fe841f959e (2026-09-18). Data as JSON: /api/errors/6647f48f373afb7c. Report an issue: GitHub.

Appendix: source

Thrown at crates/polars-stream/src/nodes/io_sinks/config.rs:123

    pub fn bytes_bufferer_config(&self) -> BytesBuffererConfig {
        if let Ok(v) = std::env::var("POLARS_IO_SINK_BYTES_BUFFERER_CONFIG") {
            (|| {
                let mut iter = v.split(',').map(|x| x.parse::<NonZeroUsize>().ok());
                let mut n = || iter.next().flatten();

                let ret = BytesBuffererConfig {
                    target_size: n()?..=n()?,
                    copy_buffer_reserve_size: n()?..=n()?,
                };

                let valid = ret.target_size.end() >= ret.target_size.start()
                    && ret.copy_buffer_reserve_size.end() >= ret.copy_buffer_reserve_size.start();

                (valid && iter.next().is_none()).then_some(ret)
            })()
            .unwrap_or_else(|| {
                panic!("invalid value for POLARS_IO_SINK_BYTES_BUFFERER_CONFIG: {v}")
            })
        } else if self.target.is_cloud_location() {
            let cloud_upload_chunk_size = self
                .cloud_upload_chunk_size()
                .unwrap_or(DEFAULT_UPLOAD_CHUNK_SIZE);

            BytesBuffererConfig {
                target_size: cloud_upload_chunk_size..=cloud_upload_chunk_size,
                copy_buffer_reserve_size: cloud_upload_chunk_size
                    .div_ceil(const { NonZeroUsize::new(8).unwrap() })
                    ..=cloud_upload_chunk_size,
            }
        } else {
            const {
                BytesBuffererConfig {
                    target_size: NonZeroUsize::new(8192).unwrap()..=NonZeroUsize::MAX,
                    copy_buffer_reserve_size: NonZeroUsize::new(8192).unwrap()..=NonZeroUsize::MAX,
                }

View on GitHub (pinned to fe841f959e)