{"record":{"id":"c132c91ebd4103e6","repo":"vllm-project/vllm","slug":"startup-handshake-timed-out-while-waiting-for-sta","errorCode":null,"errorMessage":"startup handshake timed out while waiting for {stage} after {timeout:?}","messagePattern":"startup handshake timed out while waiting for (.+?) after (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"rust/src/engine-core-client/src/error.rs","lineNumber":41,"sourceCode":"    Decode {\n        target_type: &'static str,\n        message: String,\n    },\n    #[error(\"messagepack value decode failed\")]\n    ValueDecode(#[from] rmpv::decode::Error),\n    #[error(\"messagepack ext value decode failed: {message}\")]\n    ExtValueDecode { message: String },\n    #[error(\"invalid structured outputs params: {message}\")]\n    InvalidStructuredOutputsParams { message: String },\n    #[error(\"io error\")]\n    Io(#[from] std::io::Error),\n    #[error(\"transport error\")]\n    Transport(#[from] zeromq::ZmqError),\n    #[error(\"ZMQ runtime task failed\")]\n    ZmqRuntimeTask(#[from] tokio::task::JoinError),\n    #[error(\"engine core reported fatal failure\")]\n    EngineCoreDead,\n    #[error(\"startup handshake timed out while waiting for {stage} after {timeout:?}\")]\n    HandshakeTimeout {\n        stage: &'static str,\n        timeout: Duration,\n    },\n    #[error(\"engine input registration timed out after {timeout:?}\")]\n    InputRegistrationTimeout { timeout: Duration },\n    #[error(\"unexpected engine id in startup handshake: expected {expected:?}, got {actual:?}\")]\n    UnexpectedHandshakeIdentity { expected: Vec<u8>, actual: Vec<u8> },\n    #[error(\"unexpected startup handshake message: {message}\")]\n    UnexpectedHandshakeMessage { message: String },\n    #[error(\"unexpected non-control output on coordinator path: {message}\")]\n    UnexpectedCoordinatorOutput { message: String },\n    #[error(\"unexpected output on main dispatcher path: {message}\")]\n    UnexpectedDispatcherOutput { message: String },\n    #[error(\"coordinator requires a Python-compatible two-byte engine id, got {engine_id:?}\")]\n    UnsupportedCoordinatorEngineId { engine_id: Vec<u8> },\n    #[error(\"unsupported auxiliary frame(s): expected 1 frame, got {frame_count}\")]\n    UnsupportedAuxFrames { frame_count: usize },","sourceCodeStart":23,"sourceCodeEnd":59,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/rust/src/engine-core-client/src/error.rs#L23-L59","documentation":"Error::HandshakeTimeout is raised with a stage name ('HELLO' or 'READY') and the ready_timeout duration when the startup handshake stalls. In transport.rs the frontend waits for every engine to send HELLO (line 194-199) and then READY (264-269) within ready_timeout; coordinator/bootstrap paths raise it too (coordinator/bootstrap.rs:61, mock_engine.rs:143-154).","triggerScenarios":"EngineCoreClient::connect with TransportMode::HandshakeOwner where not all engine_count engines send HELLO within ready_timeout (stage 'HELLO'), or engines do not reach READY within ready_timeout after INIT (stage 'READY', including the coordinator startup gate in step 4 of connect_handshake).","commonSituations":"EngineCount mismatch (configured engine_count larger than engines actually launched), slow model loading (large weights, cold page cache, NFS) exceeding the default timeout, engines stuck on the coordinator barrier because one engine failed, or engines unable to reach the advertised host back (wrong advertised_host in docker/K8s networking).","solutions":["Increase ready_timeout in the TransportMode::HandshakeOwner config to cover worst-case model load time","Verify the expected number of engine processes actually launched and match engine_count to it","Check stage: 'HELLO' means engines never contacted the handshake socket (network/launch issue); 'READY' means they connected but did not finish init — read engine logs","With the in-process coordinator, one stalled engine gates everyone; find the failing engine first","Confirm advertised_host is reachable from engine pods/containers"],"exampleFix":"// before\nTransportMode::HandshakeOwner { handshake_address, advertised_host, engine_count, ready_timeout: Duration::from_secs(10), .. }\n\n// after\nTransportMode::HandshakeOwner { handshake_address, advertised_host, engine_count, ready_timeout: Duration::from_secs(600), .. }","handlingStrategy":"validation","validationCode":"fn handshake_timeout_sane(engine_count: usize, ready_timeout: Duration) -> bool {\n    // cover worst-case model load: rough heuristic, tune per deployment\n    ready_timeout >= Duration::from_secs(60 * 10) && engine_count >= 1\n}","typeGuard":"fn is_handshake_timeout(e: &engine_core_client::Error) -> bool {\n    matches!(e, engine_core_client::Error::HandshakeTimeout { .. })\n}","tryCatchPattern":"match EngineCoreClient::connect(config).await {\n    Err(e @ engine_core_client::Error::HandshakeTimeout { stage, timeout }) => {\n        tracing::error!(stage, ?timeout, \"startup handshake stalled; check engine launch and reachability\");\n        // deterministic config/env issue: fix config, do not blind-retry\n    }\n    other => other?,\n}","preventionTips":["Size ready_timeout to worst-case model load time (cold cache, large weights), not average","Verify engine_count equals the number of engine processes the launcher actually starts","Confirm advertised_host resolves from engine containers before startup","With InProc coordinator, remember one stalled engine gates all READY messages"],"tags":["rust","startup","handshake","timeout","configuration"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}