xai-org/grok-build · error · io::Error

blocking pool pre-warm stalled after {started} of {n} thread

Error message

blocking pool pre-warm stalled after {started} of {n} threads

What it means

The blocking-pool pre-warm waits for a bounded time for each requested worker thread to signal readiness. If a worker parks without confirming within the timeout, the function releases the workers already started and returns io::ErrorKind::TimedOut with how many of the requested threads actually started.

Source

Thrown at crates/codegen/xai-tty-utils/src/runtime.rs:140

    drop(ready_tx);

    let deadline = Instant::now().checked_add(wait);
    let mut workers = Vec::with_capacity(n);
    for started in 0..n {
        let got = match deadline {
            Some(d) => ready_rx
                .recv_timeout(d.saturating_duration_since(Instant::now()))
                .ok(),
            None => ready_rx.recv().ok(),
        };
        match got {
            Some(thread) => workers.push(thread),
            None => {
                release_parked_workers(release, &workers);
                while let Ok(thread) = ready_rx.try_recv() {
                    thread.unpark();
                }
                return Err(io::Error::new(
                    io::ErrorKind::TimedOut,
                    format!("blocking pool pre-warm stalled after {started} of {n} threads"),
                ));
            }
        }
    }
    Ok(workers)
}

fn release_parked_workers(release: &AtomicBool, workers: &[std::thread::Thread]) {
    release.store(true, Ordering::Release);
    for thread in workers {
        thread.unpark();
    }
}

#[cfg(test)]
#[path = "runtime_tests.rs"]

View on GitHub (pinned to bc7f02eddd)

Solutions

Raise the container/host process/thread limits (cgroup pids.max, ulimit -u) and retry
Reduce the requested pre-warm thread count to match the environment's capacity
Check for a deadlock in the worker park path — if started is always 0, the ready signaling is broken

Defensive patterns

Strategy: try-catch

Try / catch

match prewarm_blocking_pool_n(n) {
    Err(e) if e.kind() == io::ErrorKind::TimedOut => {
        log::warn!("pre-warm incomplete: {e}; continuing with available workers");
        // degraded mode: proceed or retry with a smaller n
    }
    Err(e) => return Err(e.into()),
    Ok(()) => {}
}

Prevention

Raise cgroup pids.max / ulimit -u in containers that run the runtime
Size the pre-warm thread count to the environment's capacity
If pre-warm always stalls at 0 threads, audit the worker park/ready signaling for a deadlock

When it happens

Trigger: System under heavy CPU load or thread-creation limits (ulimit -u, cgroup pids.max, RLIMIT_NPROC) preventing new threads from starting or being scheduled within the pre-warm window; a misbehaving park path that never signals the ready channel.

Common situations: Containerized deployments with a low pids limit; extremely loaded CI machines; forking (run_behavior_child) right after pre-warm when resources are constrained.

Related errors

AI-assisted analysis of xai-org/grok-build@bc7f02eddd (2026-08-31). Data as JSON: /api/errors/6454b997ff3cfb77. Report an issue: GitHub.