vllm-project/vllm · critical · HandshakeError

Eager MoRIIO handshake failed for {remote_engine_id} on at l

Error message

Eager MoRIIO handshake failed for {remote_engine_id} on at least one TP rank; failing this step fast to avoid a TP collective hang

What it means

Eager handshake fans out one async handshake future per remote engine across TP ranks, then runs a CPU all-reduce (MIN) so all TP workers vote uniformly. If any single rank's handshake failed (all_ok False somewhere), the vote is 0 and this HandshakeError is raised on every rank - deliberately failing fast instead of hanging in a TP collective with divergent state.

Source

Thrown at vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py:2225

                for eid, agents in results.items():
                    self._remote_agents[eid] = agents

            logger.info(
                "Eager MoRIIO handshake: engine=%s dp_size=%d new_ranks=%d "
                "ok=%s tp_rank=%d",
                remote_engine_id,
                remote_dp_size,
                len(futures),
                all_ok,
                self.tp_rank,
            )
            # CPU all-reduce = TP-uniform success vote AND lockstep barrier: it
            # blocks until every TP worker arrives, gives them the same verdict,
            # and stays off the model compute stream.
            vote = torch.tensor([1 if all_ok else 0], device="cpu", dtype=torch.int32)
            dist.all_reduce(vote, group=self.tp_group.cpu_group, op=dist.ReduceOp.MIN)
            if int(vote.item()) == 0:
                raise HandshakeError(
                    f"Eager MoRIIO handshake failed for {remote_engine_id} on "
                    "at least one TP rank; failing this step fast to avoid a "
                    "TP collective hang"
                )

            self._eager_handshaked_engines.add(remote_engine_id)

    def start_load_kv(self, metadata: MoRIIOConnectorMetadata):
        """
        Start loading by triggering non-blocking moriio_xfer.
        We check for these trnxs to complete in each step().
        """
        self.transfer_id_to_request_id = metadata.transfer_id_to_request_id
        if self.is_producer:
            live_transfer_ids = set(self.transfer_id_to_request_id)
            self._consumer_notification_counts = {
                transfer_id: count
                for transfer_id, count in self._consumer_notification_counts.items()

View on GitHub (pinned to c794754062)

Solutions

1. Check each TP rank's logs just above this error - the rank that voted 0 logged its own handshake failure and root cause
2. Fix per-rank connectivity: firewall rules, port offsets (get_port_offset with dp/tp ranks), and peer address consistency across ranks
3. Ensure the remote engine is fully up (handshake listener bound) before eager handshake is triggered
4. Retry the step/request after the underlying per-rank issue is resolved

import socket

def all_rank_peer_ports_reachable(host: str, base_port: int, tp_size: int, dp_size: int) -> bool:
    from vllm.distributed.kv_transfer.kv_connector.v1.moriio.moriio_common import get_port_offset
    for dpr in range(dp_size):
        for tpr in range(tp_size):
            port = base_port + get_port_offset(dpr, tpr, tp_size)
            try:
                with socket.create_connection((host, port), timeout=2.0):
                    pass
            except OSError:
                return False
    return True

try:
    eager_handshake(remote_engine_id)
except HandshakeError as e:
    if "at least one TP rank" in str(e):
        # inspect per-rank logs for the rank-local root cause, fix, then retry the step
        collect_rank_local_handshake_errors(); retry_after_fix()
    else:
        raise

When it happens

Trigger: Any per-rank eager handshake failure: a port-offset collision so one rank connected to the wrong peer port, transient network failure on one node, peer ROUTER not yet listening for that rank, or any of the underlying HandshakeErrors (unexpected frame, etc.) occurring on a subset of ranks.

Common situations: Multi-node TP where one node's firewall or route blocks the peer; DP/TP port-offset miscomputation affecting one rank; starting transfers while the remote engine is still initializing; partial peer restarts.

Understand the failure class

• SSL/TLS and certificate errors — how TLS handshakes and certificate validation fail.

Related errors

• NCCL error: {error_str}
• {type(self).__name__} received pp_rank > 0 handshake metadat
• Failed to connect to metadata server: {e}
• Worker with dp_rank={payload.dp_rank}, tp_rank={payload.tp_r
• TP sizes must be positive

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/b0ebebd2f836b655. Report an issue: GitHub.