{"record":{"id":"248261979cf15132","repo":"vllm-project/vllm","slug":"engine-control-channel-closed-unexpectedly-messa","errorCode":null,"errorMessage":"engine control channel closed unexpectedly: {message}","messagePattern":"engine control channel closed unexpectedly: (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"critical","filePath":"rust/src/engine-core-client/src/error.rs","lineNumber":67,"sourceCode":"    UnexpectedHandshakeIdentity { expected: Vec<u8>, actual: Vec<u8> },\n    #[error(\"unexpected startup handshake message: {message}\")]\n    UnexpectedHandshakeMessage { message: String },\n    #[error(\"unexpected non-control output on coordinator path: {message}\")]\n    UnexpectedCoordinatorOutput { message: String },\n    #[error(\"unexpected output on main dispatcher path: {message}\")]\n    UnexpectedDispatcherOutput { message: String },\n    #[error(\"coordinator requires a Python-compatible two-byte engine id, got {engine_id:?}\")]\n    UnsupportedCoordinatorEngineId { engine_id: Vec<u8> },\n    #[error(\"unsupported auxiliary frame(s): expected 1 frame, got {frame_count}\")]\n    UnsupportedAuxFrames { frame_count: usize },\n    #[error(\"external coordinator mode is not implemented yet\")]\n    UnsupportedExternalCoordinator,\n    #[error(\"unsupported field `{field}` in {context}\")]\n    UnsupportedField {\n        context: &'static str,\n        field: &'static str,\n    },\n    #[error(\"engine control channel closed unexpectedly: {message}\")]\n    ControlClosed { message: String },\n    #[error(\"request `{request_id}` is already in flight\")]\n    DuplicateRequestId { request_id: String },\n    #[error(\n        \"data parallel rank {rank} is not connected to this frontend; connected ranks: {connected_ranks:?}\"\n    )]\n    InvalidDataParallelRank {\n        rank: u32,\n        connected_ranks: Vec<u32>,\n    },\n    #[error(\"engine-core output dispatcher closed: {message}\")]\n    DispatcherClosed { message: String },\n    #[error(\"engine-core client is closed: {message}\")]\n    ClientClosed { message: String },\n    #[error(\"request output stream for `{request_id}` closed unexpectedly\")]\n    RequestStreamClosed { request_id: String },\n    #[error(\"utility call `{method}` failed (call_id={call_id}): {message}\")]\n    UtilityCallFailed {","sourceCodeStart":49,"sourceCodeEnd":85,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/rust/src/engine-core-client/src/error.rs#L49-L85","documentation":"EngineCoreError::ControlClosed is raised when the ZMQ control channel between the frontend and the engine-core process terminates while a control operation (e.g. abort, utility call, shutdown handshake) is in flight. The `message` field carries the underlying channel error. It almost always means the engine-core process or its control socket died, so the client can no longer send commands.","triggerScenarios":"Calling any control-plane API on EngineCoreClient (abort, utility calls, shutdown) after the engine-core process has crashed, been killed (OOM, SIGKILL), or exited due to a startup failure; also when the ZMQ control socket is closed prematurely during teardown racing with in-flight calls.","commonSituations":"Engine-core OOM-killed during a large batched request; mismatched ZMQ endpoint configuration between frontend and engine-core; engine-core panicking on model load while the frontend keeps issuing control commands; CI environments killing child processes.","solutions":["Check whether the engine-core process is still alive (ps / logs) and inspect its stderr for the root-cause crash","Fix the underlying engine-core failure (OOM: reduce gpu-memory-utilization or max batch size; model load error: correct model path/config)","Verify ZMQ control and output socket endpoints match on both sides","Recreate the EngineCoreClient / restart the serving process to re-establish a healthy control channel"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"pub fn is_control_closed(e: &vllm_engine_core_client::Error) -> bool {\n    matches!(e, vllm_engine_core_client::Error::ControlClosed { .. })\n}","tryCatchPattern":"match result {\n    Ok(v) => v,\n    Err(e @ vllm_engine_core_client::Error::ControlClosed { .. }) => {\n        tracing::error!(\"engine control lost: {e}\");\n        rebuild_client_and_engine().await?;\n        return Err(e.into());\n    }\n    Err(e) => return Err(e.into()),\n}","preventionTips":["Monitor engine-core process liveness (exit status / heartbeat) and stop issuing control calls once it dies","Capture engine-core stderr to a log so root-cause crashes are diagnosable","Gate control-plane calls behind a shared shutdown flag so teardown never races in-flight commands"],"tags":["zmq","engine-core","ipc","process-crash","rust"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}