dbt-labs/dbt-core · critical

OS can't spawn worker thread

Error message

OS can't spawn worker thread: {e}

What it means

When the blocking pool needs a new worker thread and the OS refuses to create it (SpawnError::NoThreads), the runtime panics with the underlying OS error. Unlike graceful shutdown (ShuttingDown), failing to spawn due to OS resource limits is unrecoverable because the submitted closure could never run.

Solutions

1. Raise the OS thread limit: increase `ulimit -u`/RLIMIT_NPROC, or raise the container's pids.max / cgroup limit
2. Reduce blocking-pool pressure: lower max_blocking_threads and audit for thread leaks in your code
3. Check process memory — thread stack allocation can fail under memory pressure; reduce stack size or free memory
4. Ensure the runtime/pool is not being spawned repeatedly (e.g. per-request) instead of being reused

Example fix

// before
cmd.args(["--max-blocking-threads", "4096"]);
// after
// keep the pool bounded to stay under the OS thread limit
cmd.args(["--max-blocking-threads", "256"]);

// Check headroom before spawning heavy blocking work
let max_threads = std::thread::available_parallelism().ok();
// On Linux, verify the pids limit:
// let limit = std::fs::read_to_string("/sys/fs/cgroup/pids.max")?;
// let current = std::fs::read_to_string("/sys/fs/cgroup/pids.current")?;
assert!(max_threads.is_some(), "cannot gauge thread headroom");

fn can_spawn_threads(limit: usize, current: usize) -> bool { current < limit.saturating_sub(8) }

match std::panic::catch_unwind(|| pool.spawn_blocking(job)) {
    Ok(h) => h.await,
    Err(_) => { /* degrade: run job inline or return 503 */ }
}

When it happens

Trigger: Calling spawn_blocking (or any API that offloads blocking work, e.g. tokio::task::spawn_blocking routed through this pool) when the OS cannot create a thread — thread count limit reached (ulimit -u, cgroup pids.max, RLIMIT_NPROC), out of memory for a new stack, or pthread_create returning EAGAIN.

Common situations: Containerized workloads with low pids limits, running many runtimes/pools in one process, thread leaks elsewhere in the application exhausting the process thread budget, or very low `ulimit -u` on CI machines.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Adapter must be configured for the parse phase
• Adapter should be available during parse phase
• agate_table
• AgateTable exists
• All snapshot macros should start with 'snapshot_'

AI-assisted analysis of dbt-labs/dbt-core@0267ce9170 (2026-09-07). Data as JSON: /api/errors/91f23ca1d19c9ecd. Report an issue: GitHub.

Appendix: source

Thrown at crates/dbt-runtime/src/pool.rs:322

                Mandatory::NonMandatory,
                SpawnMeta::new_unnamed(fn_size),
                rt,
            )
        } else {
            self.spawn_blocking_inner(
                func,
                Mandatory::NonMandatory,
                SpawnMeta::new_unnamed(fn_size),
                rt,
            )
        };

        match spawn_result {
            Ok(()) => join_handle,
            // Compat: do not panic here, return the join_handle even though it will never resolve
            Err(SpawnError::ShuttingDown) => join_handle,
            Err(SpawnError::NoThreads(e)) => {
                panic!("OS can't spawn worker thread: {e}")
            }
        }
    }

    #[track_caller]
    pub(crate) fn spawn_mandatory_blocking<F, R>(
        &self,
        rt: &Handle,
        func: F,
    ) -> Option<JoinHandle<R>>
    where
        F: FnOnce() -> R + Send + 'static,
        R: Send + 'static,
    {
        let fn_size = size_of::<F>();
        let (join_handle, spawn_result) = if fn_size > BOX_FUTURE_THRESHOLD {
            self.spawn_blocking_inner(
                Box::new(func),

View on GitHub (pinned to 0267ce9170)