vllm-project/vllm · critical · Error

engine control channel closed unexpectedly: {message}

Error message

engine control channel closed unexpectedly: {message}

What it means

EngineCoreError::ControlClosed is raised when the ZMQ control channel between the frontend and the engine-core process terminates while a control operation (e.g. abort, utility call, shutdown handshake) is in flight. The `message` field carries the underlying channel error. It almost always means the engine-core process or its control socket died, so the client can no longer send commands.

Source

Thrown at rust/src/engine-core-client/src/error.rs:67

    UnexpectedHandshakeIdentity { expected: Vec<u8>, actual: Vec<u8> },
    #[error("unexpected startup handshake message: {message}")]
    UnexpectedHandshakeMessage { message: String },
    #[error("unexpected non-control output on coordinator path: {message}")]
    UnexpectedCoordinatorOutput { message: String },
    #[error("unexpected output on main dispatcher path: {message}")]
    UnexpectedDispatcherOutput { message: String },
    #[error("coordinator requires a Python-compatible two-byte engine id, got {engine_id:?}")]
    UnsupportedCoordinatorEngineId { engine_id: Vec<u8> },
    #[error("unsupported auxiliary frame(s): expected 1 frame, got {frame_count}")]
    UnsupportedAuxFrames { frame_count: usize },
    #[error("external coordinator mode is not implemented yet")]
    UnsupportedExternalCoordinator,
    #[error("unsupported field `{field}` in {context}")]
    UnsupportedField {
        context: &'static str,
        field: &'static str,
    },
    #[error("engine control channel closed unexpectedly: {message}")]
    ControlClosed { message: String },
    #[error("request `{request_id}` is already in flight")]
    DuplicateRequestId { request_id: String },
    #[error(
        "data parallel rank {rank} is not connected to this frontend; connected ranks: {connected_ranks:?}"
    )]
    InvalidDataParallelRank {
        rank: u32,
        connected_ranks: Vec<u32>,
    },
    #[error("engine-core output dispatcher closed: {message}")]
    DispatcherClosed { message: String },
    #[error("engine-core client is closed: {message}")]
    ClientClosed { message: String },
    #[error("request output stream for `{request_id}` closed unexpectedly")]
    RequestStreamClosed { request_id: String },
    #[error("utility call `{method}` failed (call_id={call_id}): {message}")]
    UtilityCallFailed {

View on GitHub (pinned to c794754062)

Solutions

1. Check whether the engine-core process is still alive (ps / logs) and inspect its stderr for the root-cause crash
2. Fix the underlying engine-core failure (OOM: reduce gpu-memory-utilization or max batch size; model load error: correct model path/config)
3. Verify ZMQ control and output socket endpoints match on both sides
4. Recreate the EngineCoreClient / restart the serving process to re-establish a healthy control channel

pub fn is_control_closed(e: &vllm_engine_core_client::Error) -> bool {
    matches!(e, vllm_engine_core_client::Error::ControlClosed { .. })
}

match result {
    Ok(v) => v,
    Err(e @ vllm_engine_core_client::Error::ControlClosed { .. }) => {
        tracing::error!("engine control lost: {e}");
        rebuild_client_and_engine().await?;
        return Err(e.into());
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Calling any control-plane API on EngineCoreClient (abort, utility calls, shutdown) after the engine-core process has crashed, been killed (OOM, SIGKILL), or exited due to a startup failure; also when the ZMQ control socket is closed prematurely during teardown racing with in-flight calls.

Common situations: Engine-core OOM-killed during a large batched request; mismatched ZMQ endpoint configuration between frontend and engine-core; engine-core panicking on model load while the frontend keeps issuing control commands; CI environments killing child processes.

Related errors

• engine-core output dispatcher closed: {message}
• data parallel rank {rank} is not connected to this frontend;
• request output stream for `{request_id}` closed unexpectedly
• utility call `{method}` failed (call_id={call_id}): {message
• failed to build vLLM ZMQ runtime

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/248261979cf15132. Report an issue: GitHub.