{"record":{"id":"b024d3d9177721a3","repo":"vllm-project/vllm","slug":"unexpected-frame-received-frame-b024d3","errorCode":null,"errorMessage":"unexpected frame! {received_frame = }","messagePattern":"unexpected frame! (.+?)","errorType":"exception","errorClass":"HandshakeError","httpStatus":null,"severity":"error","filePath":"vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py","lineNumber":1588,"sourceCode":"\n            if len(self.local_kv_cache_metadata) > 0:\n                logger.warning(\n                    \"len(self.local_kv_cache_metadata) = %s,\"\n                    \"maybe you didnt clear this buffer correctly\",\n                    len(self.local_kv_cache_metadata),\n                )\n                self.local_kv_cache_metadata = []\n            if len(self.remote_kv_cache_metadata) > 0:\n                logger.warning(\n                    \"len(self.remote_kv_cache_metadata) = %s,\"\n                    \"maybe you didnt clear this buffer correctly\",\n                    len(self.remote_kv_cache_metadata),\n                )\n                self.remote_kv_cache_metadata = []\n\n            received_frame = sock.recv_multipart()\n            if len(received_frame) != 2 or received_frame[0] != b\"\":\n                raise HandshakeError(f\"unexpected frame! {received_frame = }\")\n            buf = received_frame[1]\n            self.layer_name_to_remote_kv_cache_metadata[expected_engine_id] = (\n                msgpack.loads(buf)\n            )\n            self.remote_moriio_metadata[expected_engine_id] = metadata\n            setup_agent_time = time.perf_counter()\n            logger.debug(\n                \"MoRIIO handshake: add agent took: %s\",\n                setup_agent_time - got_metadata_time,\n            )\n\n        return {remote_agent_name}\n\n    def _remote_tp_rank(self, remote_tp_size: int) -> int:\n        # 0/unknown remote TP == homogeneous (avoids collapsing all ranks to 0).\n        if remote_tp_size == 0:\n            remote_tp_size = self.world_size\n        return get_moriio_remote_tp_rank(self.tp_rank, self.world_size, remote_tp_size)","sourceCodeStart":1570,"sourceCodeEnd":1606,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py#L1570-L1606","documentation":"Second stage of the DEALER handshake: after agent metadata, the client expects another two-part frame [b'', msgpack buffer] carrying per-layer KV-cache metadata for the expected engine. A frame with wrong arity or a non-empty delimiter part raises HandshakeError, indicating a desynchronized or incompatible peer stream.","triggerScenarios":"The peer sent the two handshake replies out of order, sent an error/keepalive frame in between, or the connection interleaved frames from a mismatched protocol version; also hit when the buffer was not cleared and recv alignment drifts (the code warns about uncleared remote_kv_cache_metadata buffers).","commonSituations":"vLLM version skew changing the number/order of handshake replies; reconnect races where a DEALER re-queries a busy ROUTER; leftover state from a previous failed handshake on the same socket.","solutions":["Run identical vLLM versions on producer and consumer so the handshake reply sequence matches","Ensure fresh sockets per handshake attempt (the zmq_ctx context manager already enforces teardown)","Verify only one DEALER queries one engine per transfer to avoid interleaved replies","Inspect the logged frame contents to see which message arrived where the layer-metadata buffer was expected"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    layer_meta = recv_layer_metadata(sock)\nexcept HandshakeError as e:\n    if \"unexpected frame\" in str(e):\n        # resync: redo full handshake from GET_META_MSG on a fresh socket\n        redo_handshake_fresh_socket(expected_engine_id)\n    else:\n        raise","preventionTips":["Always handshake on a fresh DEALER socket (the zmq_ctx manager ensures teardown)","Pin identical vLLM versions so reply ordering cannot drift","Clear remote metadata buffers between handshakes; heed the 'didn't clear this buffer' warning"],"tags":["zmq","handshake","protocol-mismatch","kv-transfer"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}