{"record":{"id":"91f23ca1d19c9ecd","repo":"dbt-labs/dbt-core","slug":"os-can-t-spawn-worker-thread-e","errorCode":null,"errorMessage":"OS can't spawn worker thread: {e}","messagePattern":"OS can't spawn worker thread: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"crates/dbt-runtime/src/pool.rs","lineNumber":322,"sourceCode":"                Mandatory::NonMandatory,\n                SpawnMeta::new_unnamed(fn_size),\n                rt,\n            )\n        } else {\n            self.spawn_blocking_inner(\n                func,\n                Mandatory::NonMandatory,\n                SpawnMeta::new_unnamed(fn_size),\n                rt,\n            )\n        };\n\n        match spawn_result {\n            Ok(()) => join_handle,\n            // Compat: do not panic here, return the join_handle even though it will never resolve\n            Err(SpawnError::ShuttingDown) => join_handle,\n            Err(SpawnError::NoThreads(e)) => {\n                panic!(\"OS can't spawn worker thread: {e}\")\n            }\n        }\n    }\n\n    #[track_caller]\n    pub(crate) fn spawn_mandatory_blocking<F, R>(\n        &self,\n        rt: &Handle,\n        func: F,\n    ) -> Option<JoinHandle<R>>\n    where\n        F: FnOnce() -> R + Send + 'static,\n        R: Send + 'static,\n    {\n        let fn_size = size_of::<F>();\n        let (join_handle, spawn_result) = if fn_size > BOX_FUTURE_THRESHOLD {\n            self.spawn_blocking_inner(\n                Box::new(func),","sourceCodeStart":304,"sourceCodeEnd":340,"githubUrl":"https://github.com/dbt-labs/dbt-core/blob/0267ce9170576975b76b64ce856b2e5848e96617/crates/dbt-runtime/src/pool.rs#L304-L340","documentation":"When the blocking pool needs a new worker thread and the OS refuses to create it (SpawnError::NoThreads), the runtime panics with the underlying OS error. Unlike graceful shutdown (ShuttingDown), failing to spawn due to OS resource limits is unrecoverable because the submitted closure could never run.","triggerScenarios":"Calling spawn_blocking (or any API that offloads blocking work, e.g. tokio::task::spawn_blocking routed through this pool) when the OS cannot create a thread — thread count limit reached (ulimit -u, cgroup pids.max, RLIMIT_NPROC), out of memory for a new stack, or pthread_create returning EAGAIN.","commonSituations":"Containerized workloads with low pids limits, running many runtimes/pools in one process, thread leaks elsewhere in the application exhausting the process thread budget, or very low `ulimit -u` on CI machines.","solutions":["Raise the OS thread limit: increase `ulimit -u`/RLIMIT_NPROC, or raise the container's pids.max / cgroup limit","Reduce blocking-pool pressure: lower max_blocking_threads and audit for thread leaks in your code","Check process memory — thread stack allocation can fail under memory pressure; reduce stack size or free memory","Ensure the runtime/pool is not being spawned repeatedly (e.g. per-request) instead of being reused"],"exampleFix":"// before\ncmd.args([\"--max-blocking-threads\", \"4096\"]);\n// after\n// keep the pool bounded to stay under the OS thread limit\ncmd.args([\"--max-blocking-threads\", \"256\"]);","handlingStrategy":"fallback","validationCode":"// Check headroom before spawning heavy blocking work\nlet max_threads = std::thread::available_parallelism().ok();\n// On Linux, verify the pids limit:\n// let limit = std::fs::read_to_string(\"/sys/fs/cgroup/pids.max\")?;\n// let current = std::fs::read_to_string(\"/sys/fs/cgroup/pids.current\")?;\nassert!(max_threads.is_some(), \"cannot gauge thread headroom\");","typeGuard":"fn can_spawn_threads(limit: usize, current: usize) -> bool { current < limit.saturating_sub(8) }","tryCatchPattern":"match std::panic::catch_unwind(|| pool.spawn_blocking(job)) {\n    Ok(h) => h.await,\n    Err(_) => { /* degrade: run job inline or return 503 */ }\n}","preventionTips":["Raise ulimit -u / cgroup pids.max in containers","Bound max_blocking_threads to a value well under the OS limit","Reuse a single runtime/pool process-wide instead of creating one per request","Monitor thread count and memory to catch leaks before exhaustion"],"tags":["rust","threads","os-resource-limit","panic"],"backgroundTag":"resource-not-found","analyzedSha":"0267ce9170576975b76b64ce856b2e5848e96617","analyzedAt":"2026-09-07T21:53:39.732Z","contentChangedAt":"2026-09-07T21:53:39.732Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}