vllm-project/vllm · critical · RuntimeError

Ping failed after retries

Error message

Ping failed after {retry_count} retries

What it means

The connector's keep-alive loop periodically pings the remote MoRI-IO engine over ZMQ. After MoRIIOConstants.MAX_PING_RETRIES consecutive failures (OSError or any unexpected exception per attempt) it raises RuntimeError, aborting the loop because the peer is considered unreachable.

Solutions

1. Check whether the peer vLLM instance is alive and restart it if it crashed
2. Verify network reachability of the peer host/port (telnet/nc from the pinging node) and open the ZMQ port range in the firewall
3. Confirm the peer address (host, base port, DP/TP port offsets) matches what the peer actually bound
4. If pings fail only during peer restarts, sequence restarts so producers reconnect after consumers are up

import socket

def peer_reachable(host: str, port: int, timeout: float = 2.0) -> bool:
    try:
        with socket.create_connection((host, port), timeout=timeout):
            return True
    except OSError:
        return False

# before enabling long-running ping loop:
assert peer_reachable(remote_host, handshake_port), f"peer {remote_host}:{handshake_port} unreachable"

try:
    run_ping_loop(...)
except RuntimeError as e:
    if "Ping failed" in str(e):
        # peer is down: restart/skip rather than crash the engine
        trigger_peer_health_check_and_backoff()
    else:
        raise

When it happens

Trigger: Sustained ping failure over MAX_PING_RETRIES intervals: peer process crashed or restarted, network partition, wrong host/port in the peer address, firewall dropping the connection.

Common situations: Decode instance dies while prefill keeps pinging; cross-node deployment with firewall/SecurityGroup rules blocking the ZMQ port; port offset collisions in multi-pod DP/TP layouts so pings hit nothing; NAT/DNS changes making the peer address stale.

Related errors

• Eager MoRIIO handshake failed for
• Failed to connect to metadata server
• handshake failed, unexpected msg type
• Malformed zmq_address
• tokenize endpoint unavailable

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/d0b5614ba4e51f42. Report an issue: GitHub.

Appendix: source

Thrown at vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py:1431

                        self.local_ping_port,
                        self.proxy_ip,
                        self.proxy_ping_port,
                    )
                    retry_count += 1

                except OSError as e:
                    logger.info("OS error when sending ping: %s", e)
                    retry_count += 1

                except Exception as e:
                    logger.info("Unexpected error when sending ping: %s", e)
                    retry_count += 1
                    if retry_count >= MoRIIOConstants.MAX_PING_RETRIES:
                        logger.error(
                            "Max retries (%s) exceeded. Stopping ping loop.",
                            MoRIIOConstants.MAX_PING_RETRIES,
                        )
                        raise RuntimeError(
                            f"Ping failed after {retry_count} retries"
                        ) from e

                finally:
                    time.sleep(MoRIIOConstants.PING_INTERVAL)
                    index += 1

    def shutdown(self):
        if hasattr(self, "moriio_wrapper") and self.moriio_wrapper:
            self.moriio_wrapper.shutdown()

        if hasattr(self, "_handshake_initiation_executor"):
            self._handshake_initiation_executor.shutdown(wait=False)

        if (
            hasattr(self, "_moriio_handshake_listener_t")
            and self._moriio_handshake_listener_t
        ):

View on GitHub (pinned to c794754062)