{"record":{"id":"6454b997ff3cfb77","repo":"xai-org/grok-build","slug":"blocking-pool-pre-warm-stalled-after-started-of","errorCode":null,"errorMessage":"blocking pool pre-warm stalled after {started} of {n} threads","messagePattern":"blocking pool pre-warm stalled after (.+?) of (.+?) threads","errorType":"exception","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"crates/codegen/xai-tty-utils/src/runtime.rs","lineNumber":140,"sourceCode":"    drop(ready_tx);\n\n    let deadline = Instant::now().checked_add(wait);\n    let mut workers = Vec::with_capacity(n);\n    for started in 0..n {\n        let got = match deadline {\n            Some(d) => ready_rx\n                .recv_timeout(d.saturating_duration_since(Instant::now()))\n                .ok(),\n            None => ready_rx.recv().ok(),\n        };\n        match got {\n            Some(thread) => workers.push(thread),\n            None => {\n                release_parked_workers(release, &workers);\n                while let Ok(thread) = ready_rx.try_recv() {\n                    thread.unpark();\n                }\n                return Err(io::Error::new(\n                    io::ErrorKind::TimedOut,\n                    format!(\"blocking pool pre-warm stalled after {started} of {n} threads\"),\n                ));\n            }\n        }\n    }\n    Ok(workers)\n}\n\nfn release_parked_workers(release: &AtomicBool, workers: &[std::thread::Thread]) {\n    release.store(true, Ordering::Release);\n    for thread in workers {\n        thread.unpark();\n    }\n}\n\n#[cfg(test)]\n#[path = \"runtime_tests.rs\"]","sourceCodeStart":122,"sourceCodeEnd":158,"githubUrl":"https://github.com/xai-org/grok-build/blob/bc7f02eddd3d84085849dc19ed216f11c23b0571/crates/codegen/xai-tty-utils/src/runtime.rs#L122-L158","documentation":"The blocking-pool pre-warm waits for a bounded time for each requested worker thread to signal readiness. If a worker parks without confirming within the timeout, the function releases the workers already started and returns io::ErrorKind::TimedOut with how many of the requested threads actually started.","triggerScenarios":"System under heavy CPU load or thread-creation limits (ulimit -u, cgroup pids.max, RLIMIT_NPROC) preventing new threads from starting or being scheduled within the pre-warm window; a misbehaving park path that never signals the ready channel.","commonSituations":"Containerized deployments with a low pids limit; extremely loaded CI machines; forking (run_behavior_child) right after pre-warm when resources are constrained.","solutions":["Raise the container/host process/thread limits (cgroup pids.max, ulimit -u) and retry","Reduce the requested pre-warm thread count to match the environment's capacity","Check for a deadlock in the worker park path — if started is always 0, the ready signaling is broken"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"match prewarm_blocking_pool_n(n) {\n    Err(e) if e.kind() == io::ErrorKind::TimedOut => {\n        log::warn!(\"pre-warm incomplete: {e}; continuing with available workers\");\n        // degraded mode: proceed or retry with a smaller n\n    }\n    Err(e) => return Err(e.into()),\n    Ok(()) => {}\n}","preventionTips":["Raise cgroup pids.max / ulimit -u in containers that run the runtime","Size the pre-warm thread count to the environment's capacity","If pre-warm always stalls at 0 threads, audit the worker park/ready signaling for a deadlock"],"tags":["runtime","threads","timeout","tokio","rust"],"backgroundTag":"thread-pool-prewarm-timeout","analyzedSha":"bc7f02eddd3d84085849dc19ed216f11c23b0571","analyzedAt":"2026-08-31T04:59:42.031Z","schemaVersion":2},"datasetVersion":"2026-08-31T09:17:48.483Z"}