What causes "engine core reported fatal failure" in vllm-project/vllm?

Check the engine process stderr/logs — the Rust side only reports the death, the cause is on the Python side

vllm-project/vllm · critical · Error

engine core reported fatal failure

Error message

engine core reported fatal failure

What it means

Error::EngineCoreDead signals that the Python EngineCore process reported fatal failure or its transport vanished. It is raised in transport.rs:577 (output loop sends Err(Error::EngineCoreDead) to pending requesters when the output stream ends) and in client/imp.rs:517-530 where close_registries fails every in-flight request with it. After it fires, all pending and future requests fail; the client must be rebuilt.

Source

Thrown at rust/src/engine-core-client/src/error.rs:39

    },
    #[error("messagepack decode failed for {target_type}: {message}")]
    Decode {
        target_type: &'static str,
        message: String,
    },
    #[error("messagepack value decode failed")]
    ValueDecode(#[from] rmpv::decode::Error),
    #[error("messagepack ext value decode failed: {message}")]
    ExtValueDecode { message: String },
    #[error("invalid structured outputs params: {message}")]
    InvalidStructuredOutputsParams { message: String },
    #[error("io error")]
    Io(#[from] std::io::Error),
    #[error("transport error")]
    Transport(#[from] zeromq::ZmqError),
    #[error("ZMQ runtime task failed")]
    ZmqRuntimeTask(#[from] tokio::task::JoinError),
    #[error("engine core reported fatal failure")]
    EngineCoreDead,
    #[error("startup handshake timed out while waiting for {stage} after {timeout:?}")]
    HandshakeTimeout {
        stage: &'static str,
        timeout: Duration,
    },
    #[error("engine input registration timed out after {timeout:?}")]
    InputRegistrationTimeout { timeout: Duration },
    #[error("unexpected engine id in startup handshake: expected {expected:?}, got {actual:?}")]
    UnexpectedHandshakeIdentity { expected: Vec<u8>, actual: Vec<u8> },
    #[error("unexpected startup handshake message: {message}")]
    UnexpectedHandshakeMessage { message: String },
    #[error("unexpected non-control output on coordinator path: {message}")]
    UnexpectedCoordinatorOutput { message: String },
    #[error("unexpected output on main dispatcher path: {message}")]
    UnexpectedDispatcherOutput { message: String },
    #[error("coordinator requires a Python-compatible two-byte engine id, got {engine_id:?}")]
    UnsupportedCoordinatorEngineId { engine_id: Vec<u8> },

View on GitHub (pinned to c794754062)

Solutions

Check the engine process stderr/logs — the Rust side only reports the death, the cause is on the Python side
If OOM: lower gpu-memory-utilization, max_num_seqs, or max_model_len
Restart the EngineCoreClient from scratch (connect again) — this error is terminal for the client instance
Guard load tests that kill engines intentionally by treating EngineCoreDead as the expected outcome for pending requests

Example fix

// before
if let Err(e) = outputs.next().await { /* generic handling */ }

// after
match outputs.next().await {
    Some(Err(Error::EngineCoreDead)) => {
        tracing::error!("engine died; draining in-flight requests and reconnecting");
        rebuild_client().await?;
    }
    other => { /* ... */ }
}

Defensive patterns

Strategy: fallback

Validate before calling

async fn engine_alive(client: &EngineCoreClient) -> bool {
    // cheap liveness probe using the utility/ping surface before committing new work
    client.ping().await.is_ok()
}

Type guard

fn is_engine_dead(e: &engine_core_client::Error) -> bool {
    matches!(e, engine_core_client::Error::EngineCoreDead)
}

Try / catch

match result {
    Err(e @ engine_core_client::Error::EngineCoreDead) => {
        tracing::error!("engine core fatal failure: failing over / rebuilding client");
        rebuild_client().await?; // all in-flight requests are already failed by close_registries
    }
    other => other?,
}

Prevention

Treat EngineCoreDead as terminal: rebuild the client, never retry on the same instance
Ship engine stderr to the same log aggregator so the Python-side root cause is findable
Provision engines with memory headroom (gpu-memory-utilization, max_num_seqs) to avoid OOM crashes
In tests that kill engines, assert pending requests observe EngineCoreDead (pattern in tests/client.rs:1596)

When it happens

Trigger: The engine output socket closes while requests are in flight (engine crashed, was killed, or hit OOM during CUDA init or inference), or the client detects fatal failure and sweeps all tracked requests with EngineCoreDead. Tests reproduce it by killing mock engines (tests/client.rs:1596-1606).

Common situations: CUDA OOM inside the engine, engine-side Python exception crashing the EngineCoreProc, orchestrator killing engines during scale-down, or handshake succeeding but the engine dying immediately after. The engine-side log/stderr at the same timestamp contains the actual crash cause.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/16ae7d8297f58bc7. Report an issue: GitHub.