vllm-project/vllm · error · Error

startup handshake timed out while waiting for {stage} after

Error message

startup handshake timed out while waiting for {stage} after {timeout:?}

What it means

Error::HandshakeTimeout is raised with a stage name ('HELLO' or 'READY') and the ready_timeout duration when the startup handshake stalls. In transport.rs the frontend waits for every engine to send HELLO (line 194-199) and then READY (264-269) within ready_timeout; coordinator/bootstrap paths raise it too (coordinator/bootstrap.rs:61, mock_engine.rs:143-154).

Source

Thrown at rust/src/engine-core-client/src/error.rs:41

    Decode {
        target_type: &'static str,
        message: String,
    },
    #[error("messagepack value decode failed")]
    ValueDecode(#[from] rmpv::decode::Error),
    #[error("messagepack ext value decode failed: {message}")]
    ExtValueDecode { message: String },
    #[error("invalid structured outputs params: {message}")]
    InvalidStructuredOutputsParams { message: String },
    #[error("io error")]
    Io(#[from] std::io::Error),
    #[error("transport error")]
    Transport(#[from] zeromq::ZmqError),
    #[error("ZMQ runtime task failed")]
    ZmqRuntimeTask(#[from] tokio::task::JoinError),
    #[error("engine core reported fatal failure")]
    EngineCoreDead,
    #[error("startup handshake timed out while waiting for {stage} after {timeout:?}")]
    HandshakeTimeout {
        stage: &'static str,
        timeout: Duration,
    },
    #[error("engine input registration timed out after {timeout:?}")]
    InputRegistrationTimeout { timeout: Duration },
    #[error("unexpected engine id in startup handshake: expected {expected:?}, got {actual:?}")]
    UnexpectedHandshakeIdentity { expected: Vec<u8>, actual: Vec<u8> },
    #[error("unexpected startup handshake message: {message}")]
    UnexpectedHandshakeMessage { message: String },
    #[error("unexpected non-control output on coordinator path: {message}")]
    UnexpectedCoordinatorOutput { message: String },
    #[error("unexpected output on main dispatcher path: {message}")]
    UnexpectedDispatcherOutput { message: String },
    #[error("coordinator requires a Python-compatible two-byte engine id, got {engine_id:?}")]
    UnsupportedCoordinatorEngineId { engine_id: Vec<u8> },
    #[error("unsupported auxiliary frame(s): expected 1 frame, got {frame_count}")]
    UnsupportedAuxFrames { frame_count: usize },

View on GitHub (pinned to c794754062)

Solutions

1. Increase ready_timeout in the TransportMode::HandshakeOwner config to cover worst-case model load time
2. Verify the expected number of engine processes actually launched and match engine_count to it
3. Check stage: 'HELLO' means engines never contacted the handshake socket (network/launch issue); 'READY' means they connected but did not finish init — read engine logs
4. With the in-process coordinator, one stalled engine gates everyone; find the failing engine first
5. Confirm advertised_host is reachable from engine pods/containers

Example fix

// before
TransportMode::HandshakeOwner { handshake_address, advertised_host, engine_count, ready_timeout: Duration::from_secs(10), .. }

// after
TransportMode::HandshakeOwner { handshake_address, advertised_host, engine_count, ready_timeout: Duration::from_secs(600), .. }

fn handshake_timeout_sane(engine_count: usize, ready_timeout: Duration) -> bool {
    // cover worst-case model load: rough heuristic, tune per deployment
    ready_timeout >= Duration::from_secs(60 * 10) && engine_count >= 1
}

fn is_handshake_timeout(e: &engine_core_client::Error) -> bool {
    matches!(e, engine_core_client::Error::HandshakeTimeout { .. })
}

match EngineCoreClient::connect(config).await {
    Err(e @ engine_core_client::Error::HandshakeTimeout { stage, timeout }) => {
        tracing::error!(stage, ?timeout, "startup handshake stalled; check engine launch and reachability");
        // deterministic config/env issue: fix config, do not blind-retry
    }
    other => other?,
}

When it happens

Trigger: EngineCoreClient::connect with TransportMode::HandshakeOwner where not all engine_count engines send HELLO within ready_timeout (stage 'HELLO'), or engines do not reach READY within ready_timeout after INIT (stage 'READY', including the coordinator startup gate in step 4 of connect_handshake).

Common situations: EngineCount mismatch (configured engine_count larger than engines actually launched), slow model loading (large weights, cold page cache, NFS) exceeding the default timeout, engines stuck on the coordinator barrier because one engine failed, or engines unable to reach the advertised host back (wrong advertised_host in docker/K8s networking).

Understand the failure class

• SSL/TLS and certificate errors — how TLS handshakes and certificate validation fail.
• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• managed frontend engine count ({engine_count}) must equal da
• engine input registration timed out after {timeout:?}
• unexpected engine id in startup handshake: expected {expecte
• invalid --allowed-methods value {method:?}: {e}
• invalid --allowed-headers value {header:?}: {e}

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/c132c91ebd4103e6. Report an issue: GitHub.