What causes "could not spawn threads" in pola-rs/polars?

Cap polars' thread count to fit the environment: export POLARS_MAX_THREADS=4 (or set it programmatically before first use)

pola-rs/polars · critical

could not spawn threads

Error message

could not spawn threads

What it means

polars lazily builds a global rayon thread pool sized from POLARS_MAX_THREADS / the config max_threads on first parallel use. If the OS refuses to create that many threads (process thread limit, cgroup pids limit, memory exhaustion), ThreadPoolBuilder::build fails and LazyLock panics 'could not spawn threads'.

Source

Thrown at crates/polars-core/src/runtime.rs:199

            match out.unwrap() {
                Ok(v) => v,
                Err(panic) => std::panic::resume_unwind(panic),
            }
        } else {
            f()
        }
    }
}

// this is re-exported in utils for polars child crates
#[cfg(not(target_family = "wasm"))] // only use this on non wasm targets
pub static THREAD_POOL: LazyLock<ThreadPool> = LazyLock::new(|| {
    let thread_name = std::env::var("POLARS_THREAD_NAME").unwrap_or_else(|_| "polars".to_string());
    ThreadPoolBuilder::new()
        .num_threads(polars_config::config().max_threads())
        .thread_name(move |i| format!("{thread_name}-{i}"))
        .build()
        .expect("could not spawn threads")
});

#[cfg(all(target_os = "emscripten", target_family = "wasm"))] // Use 1 rayon thread on emscripten
pub static THREAD_POOL: LazyLock<ThreadPool> = LazyLock::new(|| {
    ThreadPoolBuilder::new()
        .num_threads(1)
        .use_current_thread()
        .build()
        .expect("could not create pool")
});

pub use polars_async::ASYNC;

View on GitHub (pinned to 9b5d73fd00)

Solutions

Cap polars' thread count to fit the environment: export POLARS_MAX_THREADS=4 (or set it programmatically before first use)
Raise the OS limits: ulimit -u, container pids.max / --pids-limit, or free memory
Trigger pool initialization early (e.g. POLARS_MAX_THREADS set at process start) so the failure surfaces at a controlled point

Example fix

# before
python -c "import polars as pl; pl.len()"  # panics if thread limit hit

# after
export POLARS_MAX_THREADS=4
python -c "import polars as pl; pl.len()"

Defensive patterns

Strategy: validation

Validate before calling

# set before importing/first using polars
import os
os.environ["POLARS_MAX_THREADS"] = "4"
import polars as pl

Prevention

Set POLARS_MAX_THREADS explicitly to a value well under the environment's thread limit
Check ulimit -u and container pids.max when deploying polars in constrained environments
Initialize the pool deliberately at startup so failures surface during boot, not mid-query

When it happens

Trigger: The first parallel operation in the process (any group_by, join, or parallel iteration) in an environment that caps thread creation: ulimit -u reached, container pids.max exhausted, RLIMIT_NPROC, or POLARS_MAX_THREADS set above what the environment allows.

Common situations: Containers/Kubernetes with low pids limits; CI sandboxes; servers that already host many threads (async runtimes, other rayon pools); POLARS_MAX_THREADS misconfigured to a huge value; embedder apps that fork after polars loads.

Related errors

AI-assisted analysis of pola-rs/polars@9b5d73fd00 (2026-08-19). Data as JSON: /api/errors/8341ecaff3290fdf. Report an issue: GitHub.