vllm-project/vllm · critical

failed to build request runtime

Error message

failed to build request runtime

What it means

A panic (expect), not a Result error: build_request_runtime() calls Builder::build() on a multi-thread Tokio runtime and unwraps it. Tokio's runtime builder fails only when OS-level resource acquisition fails (thread creation blocked, ulimit exhaustion, memory pressure). This runtime exists so CPU-heavy request preparation (inference/tokenization routes) does not monopolize the HTTP runtime's workers.

Source

Thrown at rust/src/server/src/runtime.rs:23

use tracing::{info, warn};
use vllm_engine_core_client::runtime::BackgroundShutdownRuntime;

const REQUEST_WORKER_THREADS_ENV: &str = "VLLM_RS_REQUEST_WORKER_THREADS";
const DEFAULT_MAX_REQUEST_WORKER_THREADS: usize = 32;

/// Build a Tokio runtime for heavyweight request paths outside the HTTP runtime.
///
/// The server middleware uses this runtime for inference and tokenization
/// routes so CPU-heavy request preparation does not monopolize the HTTP
/// runtime's worker queue. Dropping the wrapper shuts the runtime down in the
/// background.
pub(crate) fn build_request_runtime() -> BackgroundShutdownRuntime {
    Builder::new_multi_thread()
        .enable_all()
        .thread_name("vllm-request")
        .worker_threads(request_worker_threads())
        .build()
        .expect("failed to build request runtime")
        .into()
}

/// Get the number of worker threads to use for the request runtime.
///
/// If `VLLM_RS_REQUEST_WORKER_THREADS` is set to a valid positive integer, it is
/// used directly. Otherwise, the runtime uses available parallelism capped by
/// `DEFAULT_MAX_REQUEST_WORKER_THREADS`.
fn request_worker_threads() -> usize {
    if let Some(value) = std::env::var_os(REQUEST_WORKER_THREADS_ENV) {
        match value.to_string_lossy().parse::<usize>() {
            Ok(worker_threads) if worker_threads > 0 => return worker_threads,
            _ => warn!(
                value = %value.to_string_lossy(),
                "ignoring invalid {REQUEST_WORKER_THREADS_ENV}"
            ),
        }
    }

View on GitHub (pinned to c794754062)

Solutions

1. Check thread limits: ulimit -u, and cgroup pids.max (container) — raise them or reduce thread counts.
2. Set VLLM_RS_REQUEST_WORKER_THREADS to a small positive integer (e.g. 2-4) to shrink the pool.
3. Free memory / reduce competing processes; thread stack allocation fails under OOM.
4. If it persists, reproduce with a minimal Tokio program to confirm it is an environment issue, not vLLM.

Example fix

# before: container pids.max=64 with many thread pools

# after: cap request worker threads
export VLLM_RS_REQUEST_WORKER_THREADS=4

// pre-flight in your launcher: can we still spawn threads?
let _ = std::thread::Builder::new().spawn(|| {}).map_err(|e| {
    eprintln!("cannot spawn threads: {e}");
})?;

// best-effort containment for a startup panic (the API itself offers no Result):
let rt = std::panic::catch_unwind(build_request_runtime)
    .unwrap_or_else(|_| { /* fall back to fewer threads via env, or abort with diagnostics */ });
// prefer prevention: cap VLLM_RS_REQUEST_WORKER_THREADS and raise pids limits.

When it happens

Trigger: Process starts, hits build_request_runtime(), and the OS refuses to spawn worker threads: thread count RLIMIT hit, cgroup pids.max reached, or severe memory exhaustion. Worker thread count comes from VLLM_RS_REQUEST_WORKER_THREADS if a valid positive integer, else available parallelism capped by DEFAULT_MAX_REQUEST_WORKER_THREADS.

Common situations: Containers with low pids limits or CPU quotas while available_parallelism() reports many cores; setting VLLM_RS_REQUEST_WORKER_THREADS very high alongside other thread pools; host under memory pressure during engine startup.

Related errors

• data parallel size ({}) exceeds the two-byte engine identity
• ZMQ runtime task failed
• failed to build vLLM ZMQ runtime
• invalid --allowed-methods value {method:?}: {e}
• invalid --allowed-headers value {header:?}: {e}

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/ce5c9414c6d63ba9. Report an issue: GitHub.