{"record":{"id":"ce5c9414c6d63ba9","repo":"vllm-project/vllm","slug":"failed-to-build-request-runtime","errorCode":null,"errorMessage":"failed to build request runtime","messagePattern":"failed to build request runtime","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"rust/src/server/src/runtime.rs","lineNumber":23,"sourceCode":"use tracing::{info, warn};\nuse vllm_engine_core_client::runtime::BackgroundShutdownRuntime;\n\nconst REQUEST_WORKER_THREADS_ENV: &str = \"VLLM_RS_REQUEST_WORKER_THREADS\";\nconst DEFAULT_MAX_REQUEST_WORKER_THREADS: usize = 32;\n\n/// Build a Tokio runtime for heavyweight request paths outside the HTTP runtime.\n///\n/// The server middleware uses this runtime for inference and tokenization\n/// routes so CPU-heavy request preparation does not monopolize the HTTP\n/// runtime's worker queue. Dropping the wrapper shuts the runtime down in the\n/// background.\npub(crate) fn build_request_runtime() -> BackgroundShutdownRuntime {\n    Builder::new_multi_thread()\n        .enable_all()\n        .thread_name(\"vllm-request\")\n        .worker_threads(request_worker_threads())\n        .build()\n        .expect(\"failed to build request runtime\")\n        .into()\n}\n\n/// Get the number of worker threads to use for the request runtime.\n///\n/// If `VLLM_RS_REQUEST_WORKER_THREADS` is set to a valid positive integer, it is\n/// used directly. Otherwise, the runtime uses available parallelism capped by\n/// `DEFAULT_MAX_REQUEST_WORKER_THREADS`.\nfn request_worker_threads() -> usize {\n    if let Some(value) = std::env::var_os(REQUEST_WORKER_THREADS_ENV) {\n        match value.to_string_lossy().parse::<usize>() {\n            Ok(worker_threads) if worker_threads > 0 => return worker_threads,\n            _ => warn!(\n                value = %value.to_string_lossy(),\n                \"ignoring invalid {REQUEST_WORKER_THREADS_ENV}\"\n            ),\n        }\n    }","sourceCodeStart":5,"sourceCodeEnd":41,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/rust/src/server/src/runtime.rs#L5-L41","documentation":"A panic (expect), not a Result error: build_request_runtime() calls Builder::build() on a multi-thread Tokio runtime and unwraps it. Tokio's runtime builder fails only when OS-level resource acquisition fails (thread creation blocked, ulimit exhaustion, memory pressure). This runtime exists so CPU-heavy request preparation (inference/tokenization routes) does not monopolize the HTTP runtime's workers.","triggerScenarios":"Process starts, hits build_request_runtime(), and the OS refuses to spawn worker threads: thread count RLIMIT hit, cgroup pids.max reached, or severe memory exhaustion. Worker thread count comes from VLLM_RS_REQUEST_WORKER_THREADS if a valid positive integer, else available parallelism capped by DEFAULT_MAX_REQUEST_WORKER_THREADS.","commonSituations":"Containers with low pids limits or CPU quotas while available_parallelism() reports many cores; setting VLLM_RS_REQUEST_WORKER_THREADS very high alongside other thread pools; host under memory pressure during engine startup.","solutions":["Check thread limits: ulimit -u, and cgroup pids.max (container) — raise them or reduce thread counts.","Set VLLM_RS_REQUEST_WORKER_THREADS to a small positive integer (e.g. 2-4) to shrink the pool.","Free memory / reduce competing processes; thread stack allocation fails under OOM.","If it persists, reproduce with a minimal Tokio program to confirm it is an environment issue, not vLLM."],"exampleFix":"# before: container pids.max=64 with many thread pools\n\n# after: cap request worker threads\nexport VLLM_RS_REQUEST_WORKER_THREADS=4","handlingStrategy":"try-catch","validationCode":"// pre-flight in your launcher: can we still spawn threads?\nlet _ = std::thread::Builder::new().spawn(|| {}).map_err(|e| {\n    eprintln!(\"cannot spawn threads: {e}\");\n})?;","typeGuard":null,"tryCatchPattern":"// best-effort containment for a startup panic (the API itself offers no Result):\nlet rt = std::panic::catch_unwind(build_request_runtime)\n    .unwrap_or_else(|_| { /* fall back to fewer threads via env, or abort with diagnostics */ });\n// prefer prevention: cap VLLM_RS_REQUEST_WORKER_THREADS and raise pids limits.","preventionTips":["Set VLLM_RS_REQUEST_WORKER_THREADS explicitly (small positive integer) in containers.","Check ulimit -u and cgroup pids.max before launch.","Alert on thread-count-heavy hosts; this panic is always environmental."],"tags":["panic","tokio","runtime","resources","limits","rust","vllm","startup"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}