{"record":{"id":"07820ffbe813c42e","repo":"vllm-project/vllm","slug":"managed-python-headless-engine-exited-unexpectedly","errorCode":null,"errorMessage":"managed Python headless engine exited unexpectedly with status {status}","messagePattern":"managed Python headless engine exited unexpectedly with status (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"rust/src/cmd/src/main.rs","lineNumber":209,"sourceCode":"            // Regardless of the shutdown reason, broadcast shutdown signal here to ensure\n            // that all serving tasks are notified.\n            shutdown.cancel();\n\n            // Shutdown begins. Terminate the managed engine first.\n            engine.shutdown(shutdown_timeout).await?;\n            info!(\"managed engine shut down gracefully\");\n            // Wait for the API server to shut down gracefully by draining in-flight\n            // requests.\n            if !matches!(shutdown_reason, ShutdownReason::Server(_)) {\n                serve_task.await.context(\"serve task join failed\")??;\n            }\n\n            match shutdown_reason {\n                ShutdownReason::Signal => Ok(()),\n                ShutdownReason::Server(error) => {\n                    Err(error.context(\"OpenAI server shut down unexpectedly\"))\n                }\n                ShutdownReason::EngineExited(status) => Err(anyhow!(\n                    \"managed Python headless engine exited unexpectedly with status {status}\"\n                )),\n            }\n        }\n        Command::Render(args) => {\n            vllm_server::serve_render(args.into_config(), shutdown_signal()).await\n        }\n    }\n}\n","sourceCodeStart":191,"sourceCodeEnd":219,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/rust/src/cmd/src/main.rs#L191-L219","documentation":"Managed-engine supervisor error (`cmd/src/main.rs:209`): in a `--headless`-style deployment the Rust frontend supervises Python engine processes; the `engine.wait_for_exit()` branch fired, meaning the managed Python engine process terminated (status in the message) before a shutdown was requested. The frontend then shuts everything down and returns this error (note: `ShutdownReason::Signal` returns Ok, so this error only appears for unsolicited engine death).","triggerScenarios":"The Python engine process crashes (CUDA OOM, segfault in a kernel, Python exception at startup), is OOM-killed by the kernel, or exits nonzero/negative-signal during normal serving; the Rust supervisor observes the exit and reports the raw wait status.","commonSituations":"GPU OOM or NCCL failures inside the engine; startup failures like missing model weights or bad engine args when the engine is spawned detached; host memory pressure triggering the OS OOM killer (status often shows signal 9); incompatible PyTorch/CUDA versions crashing the worker.","solutions":["Decode `status`: exit code N = engine main returned N; `signal: 9` = OOM-killed (free host RAM / lower gpu-memory-utilization); negative signals = crash (get the Python backtrace from engine logs).","Read the managed Python engine's own log/stderr — the root cause (CUDA OOM, import error, NCCL timeout) is there, not in the Rust log.","For OOM kills: reduce batch/memory settings, max_model_len, or enable tensor parallelism; increase host memory limits in Kubernetes.","For startup crashes: run the engine command manually with the same args to see the Python traceback, then fix the engine-side config.","Add liveness probes / restart policy so an engine death is auto-recovered in production."],"exampleFix":"# before\n# status: exit code: 137 / signal 9 (SIGKILL) — host OOM killer hit the engine\n\n# after\n# lower engine memory footprint so the OOM killer does not target it:\nvllm serve meta-llama/Llama-3.1-8B --max-model-len 8192 --gpu-memory-utilization 0.85","handlingStrategy":"retry","validationCode":"// pre-flight: run the engine's startup path once before serving traffic\nlet status = engine.spawn_and_handshake(handshake_timeout).await?; // fails fast with the engine's own error\nif !status.ready() { return Err(\"engine failed startup checks\"); }","typeGuard":null,"tryCatchPattern":"match run_result {\n    Err(e) if e.to_string().contains(\"managed Python headless engine exited unexpectedly\") => {\n        // decode status (exit code vs signal 9 = OOM-kill), read engine logs, restart with backoff\n        supervisor.restart_engine_with_backoff().await?;\n        retry_serve().await\n    }\n    other => other?,\n}","preventionTips":["Run a startup handshake/health gate before routing traffic so startup crashes surface with the engine's traceback.","Cap engine memory (gpu-memory-utilization, max-model-len, batch limits) and reserve host RAM to avoid OOM kills.","Ship engine logs to a central sink — the root cause of this error lives in the Python process, not the Rust supervisor.","Add liveness probes + restart policy in Kubernetes for the engine process."],"tags":["rust","engine","process-exit","oom","cuda","supervisor"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}