{"record":{"id":"b38a5ee3919df4fc","repo":"vllm-project/vllm","slug":"zmq-runtime-task-failed","errorCode":null,"errorMessage":"ZMQ runtime task failed","messagePattern":"ZMQ runtime task failed","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"rust/src/engine-core-client/src/error.rs","lineNumber":37,"sourceCode":"        target_type: &'static str,\n        message: String,\n    },\n    #[error(\"messagepack decode failed for {target_type}: {message}\")]\n    Decode {\n        target_type: &'static str,\n        message: String,\n    },\n    #[error(\"messagepack value decode failed\")]\n    ValueDecode(#[from] rmpv::decode::Error),\n    #[error(\"messagepack ext value decode failed: {message}\")]\n    ExtValueDecode { message: String },\n    #[error(\"invalid structured outputs params: {message}\")]\n    InvalidStructuredOutputsParams { message: String },\n    #[error(\"io error\")]\n    Io(#[from] std::io::Error),\n    #[error(\"transport error\")]\n    Transport(#[from] zeromq::ZmqError),\n    #[error(\"ZMQ runtime task failed\")]\n    ZmqRuntimeTask(#[from] tokio::task::JoinError),\n    #[error(\"engine core reported fatal failure\")]\n    EngineCoreDead,\n    #[error(\"startup handshake timed out while waiting for {stage} after {timeout:?}\")]\n    HandshakeTimeout {\n        stage: &'static str,\n        timeout: Duration,\n    },\n    #[error(\"engine input registration timed out after {timeout:?}\")]\n    InputRegistrationTimeout { timeout: Duration },\n    #[error(\"unexpected engine id in startup handshake: expected {expected:?}, got {actual:?}\")]\n    UnexpectedHandshakeIdentity { expected: Vec<u8>, actual: Vec<u8> },\n    #[error(\"unexpected startup handshake message: {message}\")]\n    UnexpectedHandshakeMessage { message: String },\n    #[error(\"unexpected non-control output on coordinator path: {message}\")]\n    UnexpectedCoordinatorOutput { message: String },\n    #[error(\"unexpected output on main dispatcher path: {message}\")]\n    UnexpectedDispatcherOutput { message: String },","sourceCodeStart":19,"sourceCodeEnd":55,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/rust/src/engine-core-client/src/error.rs#L19-L55","documentation":"Error::ZmqRuntimeTask wraps tokio::task::JoinError via #[from]. The client spawns background tasks (output pump, dispatcher, abort handler — the AbortOnDropHandle fields on EngineCoreClient) to drive ZMQ I/O; if one of those tasks panics or is cancelled, awaiting its join handle yields JoinError, converted into this variant.","triggerScenarios":"Awaiting a JoinHandle of one of the client's background ZMQ tasks after the task panicked (e.g. an internal expect/unwrap failed inside transport.rs, such as the 'handshake router messages must contain identity and payload' expect at transport.rs:454) or was cancelled at runtime shutdown.","commonSituations":"An internal invariant in a background task broke — usually a downstream symptom of receiving a malformed ZMQ multipart message. Also appears when the BackgroundShutdownRuntime tears down while tasks are still being joined. The panic message in the JoinError payload is the real diagnostic.","solutions":["Look at the JoinError contents: is_panic() → read the panic payload for the root cause; is_cancelled() → audit shutdown ordering","Enable RUST_BACKTRACE=1 and RUST_LOG=engine_core_client=debug to capture the panicking task's context","If caused by malformed frames, fix the upstream framing issue (see Decode/UnexpectedDispatcherOutput errors)","Check for version mismatch if the panic references message shapes"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"fn is_zmq_task_failure(e: &engine_core_client::Error) -> bool {\n    matches!(e, engine_core_client::Error::ZmqRuntimeTask(_))\n}","tryCatchPattern":"if let engine_core_client::Error::ZmqRuntimeTask(join) = &err {\n    if join.is_panic() {\n        tracing::error!(panic = ?join.into_panic(), \"background zmq task panicked; client must be rebuilt\");\n    }\n}","preventionTips":["Run with RUST_BACKTRACE=1 so task panics carry stack traces into JoinError diagnostics","Treat ZmqRuntimeTask as fatal for the client instance; rebuild rather than reuse","Keep background-task invariants (frame shapes) covered by tests to prevent the panics that surface here"],"tags":["rust","tokio","zeromq","tasks","panic"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}