pola-rs/polars · error

in-flight byte budget init must be larger than the target_ch

Error message

in-flight byte budget init must be larger than the target_chunk_size

What it means

Startup sanity check of the cloud download concurrency budget: POLARS_INFLIGHT_INIT_BYTE_BUDGET was set smaller than the target chunk size, which would deadlock the pipeline (a single request could never fit in the budget). Raise the budget above the chunk size.

Source

Thrown at crates/polars-io/src/cloud/concurrency/mod.rs:122

    let init_byte_budget = std::env::var("POLARS_INFLIGHT_INIT_BYTE_BUDGET")
        .map(|x| {
            x.parse::<NonZeroU64>()
                .unwrap_or_else(|_| {
                    panic!("invalid value for POLARS_INFLIGHT_INIT_BYTE_BUDGET: {x}")
                })
                .get()
        })
        .unwrap_or_else(|_| {
            // This should be lower than the expected BDP so it can ramp-up, but
            // too low a value delays the transition to stable.
            // Heuristic: higher bandwidth is expected on larger instances.
            let n = polars_config::config().max_threads() as u64;
            n.div_ceil(8).max(4) * target_chunk_size
        })
        .max(1);

    if init_byte_budget < target_chunk_size {
        panic!("in-flight byte budget init must be larger than the target_chunk_size");
    }

    init_byte_budget
}

/// Maximum number of requests concurrently in flight.
pub fn get_request_budget() -> u64 {
    // Since object_store/reqwest use HTTP/1 with a connection pool, this value controls the
    // max concurrent TCP sessions to S3 for the pipeline.
    // When modifying this value, consider the max_thread count configuration(s), the OS limitations
    // (e.g., ulimit -n), and any cloud infrastructure limitations.
    std::env::var("POLARS_INFLIGHT_REQUEST_BUDGET")
        .map(|x| {
            x.parse::<NonZeroU64>()
                .unwrap_or_else(|_| panic!("invalid value for POLARS_INFLIGHT_REQUEST_BUDGET: {x}"))
                .get()
        })
        .unwrap_or(512)

View on GitHub (pinned to 68506541d2)

Solutions

1. Set POLARS_INFLIGHT_INIT_BYTE_BUDGET larger than the configured target_chunk_size.
2. Lower the target chunk size or raise the initial byte budget so init > target_chunk_size.

When it happens

Trigger: This panic/expect fires when execution reaches an unguarded state described by: "in-flight byte budget init must be larger than the target_chunk_size". Typical triggers: unsupported dtype or feature-gated code path reached at runtime, invalid user input or environment variable value, calling API methods in the wrong order or on mismatched types, or data (lengths, offsets, ranges) violating the function's preconditions.

Common situations: Encountered when input data or configuration does not meet the preconditions of the throwing code path ("in-flight byte budget init must be larger than the target_chunk_size"). Common cases: missing Cargo feature flags, mismatched dtypes/lengths between arrays or series, out-of-range temporal values, malformed environment variables, or operations on unsupported/complex nested types.

AI-assisted analysis of pola-rs/polars@68506541d2 (2026-08-19). Data as JSON: /api/errors/d1d06d8332b0c5e9. Report an issue: GitHub.