{"record":{"id":"b0ebebd2f836b655","repo":"vllm-project/vllm","slug":"eager-moriio-handshake-failed-for-remote-engine-i","errorCode":null,"errorMessage":"Eager MoRIIO handshake failed for {remote_engine_id} on at least one TP rank; failing this step fast to avoid a TP collective hang","messagePattern":"Eager MoRIIO handshake failed for (.+?) on at least one TP rank; failing this step fast to avoid a TP collective hang","errorType":"exception","errorClass":"HandshakeError","httpStatus":null,"severity":"critical","filePath":"vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py","lineNumber":2225,"sourceCode":"                for eid, agents in results.items():\n                    self._remote_agents[eid] = agents\n\n            logger.info(\n                \"Eager MoRIIO handshake: engine=%s dp_size=%d new_ranks=%d \"\n                \"ok=%s tp_rank=%d\",\n                remote_engine_id,\n                remote_dp_size,\n                len(futures),\n                all_ok,\n                self.tp_rank,\n            )\n            # CPU all-reduce = TP-uniform success vote AND lockstep barrier: it\n            # blocks until every TP worker arrives, gives them the same verdict,\n            # and stays off the model compute stream.\n            vote = torch.tensor([1 if all_ok else 0], device=\"cpu\", dtype=torch.int32)\n            dist.all_reduce(vote, group=self.tp_group.cpu_group, op=dist.ReduceOp.MIN)\n            if int(vote.item()) == 0:\n                raise HandshakeError(\n                    f\"Eager MoRIIO handshake failed for {remote_engine_id} on \"\n                    \"at least one TP rank; failing this step fast to avoid a \"\n                    \"TP collective hang\"\n                )\n\n            self._eager_handshaked_engines.add(remote_engine_id)\n\n    def start_load_kv(self, metadata: MoRIIOConnectorMetadata):\n        \"\"\"\n        Start loading by triggering non-blocking moriio_xfer.\n        We check for these trnxs to complete in each step().\n        \"\"\"\n        self.transfer_id_to_request_id = metadata.transfer_id_to_request_id\n        if self.is_producer:\n            live_transfer_ids = set(self.transfer_id_to_request_id)\n            self._consumer_notification_counts = {\n                transfer_id: count\n                for transfer_id, count in self._consumer_notification_counts.items()","sourceCodeStart":2207,"sourceCodeEnd":2243,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py#L2207-L2243","documentation":"Eager handshake fans out one async handshake future per remote engine across TP ranks, then runs a CPU all-reduce (MIN) so all TP workers vote uniformly. If any single rank's handshake failed (all_ok False somewhere), the vote is 0 and this HandshakeError is raised on every rank - deliberately failing fast instead of hanging in a TP collective with divergent state.","triggerScenarios":"Any per-rank eager handshake failure: a port-offset collision so one rank connected to the wrong peer port, transient network failure on one node, peer ROUTER not yet listening for that rank, or any of the underlying HandshakeErrors (unexpected frame, etc.) occurring on a subset of ranks.","commonSituations":"Multi-node TP where one node's firewall or route blocks the peer; DP/TP port-offset miscomputation affecting one rank; starting transfers while the remote engine is still initializing; partial peer restarts.","solutions":["Check each TP rank's logs just above this error - the rank that voted 0 logged its own handshake failure and root cause","Fix per-rank connectivity: firewall rules, port offsets (get_port_offset with dp/tp ranks), and peer address consistency across ranks","Ensure the remote engine is fully up (handshake listener bound) before eager handshake is triggered","Retry the step/request after the underlying per-rank issue is resolved"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"import socket\n\ndef all_rank_peer_ports_reachable(host: str, base_port: int, tp_size: int, dp_size: int) -> bool:\n    from vllm.distributed.kv_transfer.kv_connector.v1.moriio.moriio_common import get_port_offset\n    for dpr in range(dp_size):\n        for tpr in range(tp_size):\n            port = base_port + get_port_offset(dpr, tpr, tp_size)\n            try:\n                with socket.create_connection((host, port), timeout=2.0):\n                    pass\n            except OSError:\n                return False\n    return True","typeGuard":null,"tryCatchPattern":"try:\n    eager_handshake(remote_engine_id)\nexcept HandshakeError as e:\n    if \"at least one TP rank\" in str(e):\n        # inspect per-rank logs for the rank-local root cause, fix, then retry the step\n        collect_rank_local_handshake_errors(); retry_after_fix()\n    else:\n        raise","preventionTips":["Verify every rank's offset port is reachable from every node before enabling eager handshake","Wait for the remote engine's handshake listener readiness signal before triggering","Aggregate per-rank handshake results into monitoring so the failing rank is identified instantly"],"tags":["handshake","tensor-parallel","network","kv-transfer","distributed"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}