vllm-project/vllm · error · HandshakeError

unexpected frame! {received_frame = }

Error message

unexpected frame! {received_frame = }

What it means

Second stage of the DEALER handshake: after agent metadata, the client expects another two-part frame [b'', msgpack buffer] carrying per-layer KV-cache metadata for the expected engine. A frame with wrong arity or a non-empty delimiter part raises HandshakeError, indicating a desynchronized or incompatible peer stream.

Source

Thrown at vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py:1588

            if len(self.local_kv_cache_metadata) > 0:
                logger.warning(
                    "len(self.local_kv_cache_metadata) = %s,"
                    "maybe you didnt clear this buffer correctly",
                    len(self.local_kv_cache_metadata),
                )
                self.local_kv_cache_metadata = []
            if len(self.remote_kv_cache_metadata) > 0:
                logger.warning(
                    "len(self.remote_kv_cache_metadata) = %s,"
                    "maybe you didnt clear this buffer correctly",
                    len(self.remote_kv_cache_metadata),
                )
                self.remote_kv_cache_metadata = []

            received_frame = sock.recv_multipart()
            if len(received_frame) != 2 or received_frame[0] != b"":
                raise HandshakeError(f"unexpected frame! {received_frame = }")
            buf = received_frame[1]
            self.layer_name_to_remote_kv_cache_metadata[expected_engine_id] = (
                msgpack.loads(buf)
            )
            self.remote_moriio_metadata[expected_engine_id] = metadata
            setup_agent_time = time.perf_counter()
            logger.debug(
                "MoRIIO handshake: add agent took: %s",
                setup_agent_time - got_metadata_time,
            )

        return {remote_agent_name}

    def _remote_tp_rank(self, remote_tp_size: int) -> int:
        # 0/unknown remote TP == homogeneous (avoids collapsing all ranks to 0).
        if remote_tp_size == 0:
            remote_tp_size = self.world_size
        return get_moriio_remote_tp_rank(self.tp_rank, self.world_size, remote_tp_size)

View on GitHub (pinned to c794754062)

Solutions

1. Run identical vLLM versions on producer and consumer so the handshake reply sequence matches
2. Ensure fresh sockets per handshake attempt (the zmq_ctx context manager already enforces teardown)
3. Verify only one DEALER queries one engine per transfer to avoid interleaved replies
4. Inspect the logged frame contents to see which message arrived where the layer-metadata buffer was expected

try:
    layer_meta = recv_layer_metadata(sock)
except HandshakeError as e:
    if "unexpected frame" in str(e):
        # resync: redo full handshake from GET_META_MSG on a fresh socket
        redo_handshake_fresh_socket(expected_engine_id)
    else:
        raise

When it happens

Trigger: The peer sent the two handshake replies out of order, sent an error/keepalive frame in between, or the connection interleaved frames from a mismatched protocol version; also hit when the buffer was not cleared and recv alignment drifts (the code warns about uncleared remote_kv_cache_metadata buffers).

Common situations: vLLM version skew changing the number/order of handshake replies; reconnect races where a DEALER re-queries a busy ROUTER; leftover state from a previous failed handshake on the same socket.

Related errors

• handshake failed, unexpected msg type
• Unexpected frame! {received_frame = }
• tokenize endpoint unavailable: {0}
• {type(self).__name__} received pp_rank > 0 handshake metadat
• Malformed zmq_address {zmq_address!r}: expected 'host:IP,han

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/b024d3d9177721a3. Report an issue: GitHub.