vllm-project/vllm · error · Error

ZMQ runtime task failed

Error message

ZMQ runtime task failed

What it means

Error::ZmqRuntimeTask wraps tokio::task::JoinError via #[from]. The client spawns background tasks (output pump, dispatcher, abort handler — the AbortOnDropHandle fields on EngineCoreClient) to drive ZMQ I/O; if one of those tasks panics or is cancelled, awaiting its join handle yields JoinError, converted into this variant.

Source

Thrown at rust/src/engine-core-client/src/error.rs:37

        target_type: &'static str,
        message: String,
    },
    #[error("messagepack decode failed for {target_type}: {message}")]
    Decode {
        target_type: &'static str,
        message: String,
    },
    #[error("messagepack value decode failed")]
    ValueDecode(#[from] rmpv::decode::Error),
    #[error("messagepack ext value decode failed: {message}")]
    ExtValueDecode { message: String },
    #[error("invalid structured outputs params: {message}")]
    InvalidStructuredOutputsParams { message: String },
    #[error("io error")]
    Io(#[from] std::io::Error),
    #[error("transport error")]
    Transport(#[from] zeromq::ZmqError),
    #[error("ZMQ runtime task failed")]
    ZmqRuntimeTask(#[from] tokio::task::JoinError),
    #[error("engine core reported fatal failure")]
    EngineCoreDead,
    #[error("startup handshake timed out while waiting for {stage} after {timeout:?}")]
    HandshakeTimeout {
        stage: &'static str,
        timeout: Duration,
    },
    #[error("engine input registration timed out after {timeout:?}")]
    InputRegistrationTimeout { timeout: Duration },
    #[error("unexpected engine id in startup handshake: expected {expected:?}, got {actual:?}")]
    UnexpectedHandshakeIdentity { expected: Vec<u8>, actual: Vec<u8> },
    #[error("unexpected startup handshake message: {message}")]
    UnexpectedHandshakeMessage { message: String },
    #[error("unexpected non-control output on coordinator path: {message}")]
    UnexpectedCoordinatorOutput { message: String },
    #[error("unexpected output on main dispatcher path: {message}")]
    UnexpectedDispatcherOutput { message: String },

View on GitHub (pinned to c794754062)

Solutions

1. Look at the JoinError contents: is_panic() → read the panic payload for the root cause; is_cancelled() → audit shutdown ordering
2. Enable RUST_BACKTRACE=1 and RUST_LOG=engine_core_client=debug to capture the panicking task's context
3. If caused by malformed frames, fix the upstream framing issue (see Decode/UnexpectedDispatcherOutput errors)
4. Check for version mismatch if the panic references message shapes

fn is_zmq_task_failure(e: &engine_core_client::Error) -> bool {
    matches!(e, engine_core_client::Error::ZmqRuntimeTask(_))
}

if let engine_core_client::Error::ZmqRuntimeTask(join) = &err {
    if join.is_panic() {
        tracing::error!(panic = ?join.into_panic(), "background zmq task panicked; client must be rebuilt");
    }
}

When it happens

Trigger: Awaiting a JoinHandle of one of the client's background ZMQ tasks after the task panicked (e.g. an internal expect/unwrap failed inside transport.rs, such as the 'handshake router messages must contain identity and payload' expect at transport.rs:454) or was cancelled at runtime shutdown.

Common situations: An internal invariant in a background task broke — usually a downstream symptom of receiving a malformed ZMQ multipart message. Also appears when the BackgroundShutdownRuntime tears down while tasks are still being joined. The panic message in the JoinError payload is the real diagnostic.

Related errors

• failed to build vLLM ZMQ runtime
• failed to build request runtime
• tool response messages require a tool_call_id; use ChatMessa
• transport error
• engine core reported fatal failure

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/b38a5ee3919df4fc. Report an issue: GitHub.