{"record":{"id":"16ae7d8297f58bc7","repo":"vllm-project/vllm","slug":"engine-core-reported-fatal-failure","errorCode":null,"errorMessage":"engine core reported fatal failure","messagePattern":"engine core reported fatal failure","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"critical","filePath":"rust/src/engine-core-client/src/error.rs","lineNumber":39,"sourceCode":"    },\n    #[error(\"messagepack decode failed for {target_type}: {message}\")]\n    Decode {\n        target_type: &'static str,\n        message: String,\n    },\n    #[error(\"messagepack value decode failed\")]\n    ValueDecode(#[from] rmpv::decode::Error),\n    #[error(\"messagepack ext value decode failed: {message}\")]\n    ExtValueDecode { message: String },\n    #[error(\"invalid structured outputs params: {message}\")]\n    InvalidStructuredOutputsParams { message: String },\n    #[error(\"io error\")]\n    Io(#[from] std::io::Error),\n    #[error(\"transport error\")]\n    Transport(#[from] zeromq::ZmqError),\n    #[error(\"ZMQ runtime task failed\")]\n    ZmqRuntimeTask(#[from] tokio::task::JoinError),\n    #[error(\"engine core reported fatal failure\")]\n    EngineCoreDead,\n    #[error(\"startup handshake timed out while waiting for {stage} after {timeout:?}\")]\n    HandshakeTimeout {\n        stage: &'static str,\n        timeout: Duration,\n    },\n    #[error(\"engine input registration timed out after {timeout:?}\")]\n    InputRegistrationTimeout { timeout: Duration },\n    #[error(\"unexpected engine id in startup handshake: expected {expected:?}, got {actual:?}\")]\n    UnexpectedHandshakeIdentity { expected: Vec<u8>, actual: Vec<u8> },\n    #[error(\"unexpected startup handshake message: {message}\")]\n    UnexpectedHandshakeMessage { message: String },\n    #[error(\"unexpected non-control output on coordinator path: {message}\")]\n    UnexpectedCoordinatorOutput { message: String },\n    #[error(\"unexpected output on main dispatcher path: {message}\")]\n    UnexpectedDispatcherOutput { message: String },\n    #[error(\"coordinator requires a Python-compatible two-byte engine id, got {engine_id:?}\")]\n    UnsupportedCoordinatorEngineId { engine_id: Vec<u8> },","sourceCodeStart":21,"sourceCodeEnd":57,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/rust/src/engine-core-client/src/error.rs#L21-L57","documentation":"Error::EngineCoreDead signals that the Python EngineCore process reported fatal failure or its transport vanished. It is raised in transport.rs:577 (output loop sends Err(Error::EngineCoreDead) to pending requesters when the output stream ends) and in client/imp.rs:517-530 where close_registries fails every in-flight request with it. After it fires, all pending and future requests fail; the client must be rebuilt.","triggerScenarios":"The engine output socket closes while requests are in flight (engine crashed, was killed, or hit OOM during CUDA init or inference), or the client detects fatal failure and sweeps all tracked requests with EngineCoreDead. Tests reproduce it by killing mock engines (tests/client.rs:1596-1606).","commonSituations":"CUDA OOM inside the engine, engine-side Python exception crashing the EngineCoreProc, orchestrator killing engines during scale-down, or handshake succeeding but the engine dying immediately after. The engine-side log/stderr at the same timestamp contains the actual crash cause.","solutions":["Check the engine process stderr/logs — the Rust side only reports the death, the cause is on the Python side","If OOM: lower gpu-memory-utilization, max_num_seqs, or max_model_len","Restart the EngineCoreClient from scratch (connect again) — this error is terminal for the client instance","Guard load tests that kill engines intentionally by treating EngineCoreDead as the expected outcome for pending requests"],"exampleFix":"// before\nif let Err(e) = outputs.next().await { /* generic handling */ }\n\n// after\nmatch outputs.next().await {\n    Some(Err(Error::EngineCoreDead)) => {\n        tracing::error!(\"engine died; draining in-flight requests and reconnecting\");\n        rebuild_client().await?;\n    }\n    other => { /* ... */ }\n}","handlingStrategy":"fallback","validationCode":"async fn engine_alive(client: &EngineCoreClient) -> bool {\n    // cheap liveness probe using the utility/ping surface before committing new work\n    client.ping().await.is_ok()\n}","typeGuard":"fn is_engine_dead(e: &engine_core_client::Error) -> bool {\n    matches!(e, engine_core_client::Error::EngineCoreDead)\n}","tryCatchPattern":"match result {\n    Err(e @ engine_core_client::Error::EngineCoreDead) => {\n        tracing::error!(\"engine core fatal failure: failing over / rebuilding client\");\n        rebuild_client().await?; // all in-flight requests are already failed by close_registries\n    }\n    other => other?,\n}","preventionTips":["Treat EngineCoreDead as terminal: rebuild the client, never retry on the same instance","Ship engine stderr to the same log aggregator so the Python-side root cause is findable","Provision engines with memory headroom (gpu-memory-utilization, max_num_seqs) to avoid OOM crashes","In tests that kill engines, assert pending requests observe EngineCoreDead (pattern in tests/client.rs:1596)"],"tags":["rust","engine-crash","lifecycle","zeromq"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}