vllm-project/vllm · error · HandshakeError

handshake failed, unexpected msg type

Error message

handshake failed, unexpected msg type

What it means

The handshake ROUTER listener only accepts two message types from peers: GET_META_MSG (metadata query) and POP_DONE_RECV (transfer-done notification). Any other first frame raises HandshakeError, aborting the listener thread and thus the connector's handshake capability.

Source

Thrown at vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py:1492

            "Size of encoded MoRIIOAgentMetadata: %s bytes", str(size_in_bytes)
        )

        # Listen for new requests for metadata.
        host = "*"

        path = make_zmq_path("tcp", host, base_port)
        logger.debug("mori handshake starting listening on path: %s", path)

        with zmq_ctx(zmq.ROUTER, path) as sock:
            ready_event.set()
            while True:
                identity, msg = sock.recv_multipart()
                if (
                    msg != MoRIIOConstants.GET_META_MSG
                    and msg != MoRIIOConstants.POP_DONE_RECV
                ):
                    logger.error("Connection listener got unexpected message")
                    raise HandshakeError("handshake failed, unexpected msg type")
                elif msg == MoRIIOConstants.GET_META_MSG:
                    sock.send_multipart(
                        (identity, b"", encoded_data)
                    )  # send local mori io engine meta data
                    logger.debug("MoRIIO handshake listener sent metadata")
                    # now we send tensor meta data for each block
                    buf = msgpack.dumps(layer_name_to_local_kv_cache_metadata)
                    sock.send_multipart((identity, b"", buf))
                elif msg == MoRIIOConstants.POP_DONE_RECV:
                    _, req_id = sock.recv_multipart()
                    logger.debug(
                        "MoRIIO handshake listener received done recv for req",
                        req_id.decode(),
                    )

    def _moriio_handshake(
        self,
        host: str,

View on GitHub (pinned to c794754062)

Solutions

1. Pin both P and D instances to the same vLLM version so protocol constants match
2. Verify the connecting peer is a MoRI-IO instance and computes the same port offset (dp_rank/tp_rank based)
3. Reserve disjoint port ranges per rank/instance to avoid cross-connections
4. Inspect the logged 'Connection listener got unexpected message' frame bytes to identify which client sent it

# Nothing to validate locally; verify peer identity/version before connecting.
def peer_version_matches(remote_version: str, local_version: str) -> bool:
    return remote_version == local_version

try:
    run_handshake_listener(...)
except HandshakeError as e:
    if "unexpected msg type" in str(e):
        log_offending_frame_and_peer_identity()
        restart_listener_after_peer_version_check()
    else:
        raise

When it happens

Trigger: A client connecting to the handshake port and sending an unexpected/undecodable frame: vLLM version skew where constant values differ, a non-MoRI-IO ZMQ client hitting the port, or stream corruption delivering garbage bytes.

Common situations: Mixed vLLM versions on prefill and decode with divergent MoRI-IO protocol constants; port reuse where another service or another DP/TP rank's socket connects to the wrong listener; manual ZMQ probing against the handshake port.

Understand the failure class

• SSL/TLS and certificate errors — how TLS handshakes and certificate validation fail.

Related errors

• Unexpected frame! {received_frame = }
• unexpected frame! {received_frame = }
• tokenize endpoint unavailable: {0}
• {type(self).__name__} received pp_rank > 0 handshake metadat
• Malformed zmq_address {zmq_address!r}: expected 'host:IP,han

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/55c8fb7abc9337b7. Report an issue: GitHub.