{"record":{"id":"d0b5614ba4e51f42","repo":"vllm-project/vllm","slug":"ping-failed-after-retry-count-retries","errorCode":null,"errorMessage":"Ping failed after {retry_count} retries","messagePattern":"Ping failed after (.+?) retries","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py","lineNumber":1431,"sourceCode":"                        self.local_ping_port,\n                        self.proxy_ip,\n                        self.proxy_ping_port,\n                    )\n                    retry_count += 1\n\n                except OSError as e:\n                    logger.info(\"OS error when sending ping: %s\", e)\n                    retry_count += 1\n\n                except Exception as e:\n                    logger.info(\"Unexpected error when sending ping: %s\", e)\n                    retry_count += 1\n                    if retry_count >= MoRIIOConstants.MAX_PING_RETRIES:\n                        logger.error(\n                            \"Max retries (%s) exceeded. Stopping ping loop.\",\n                            MoRIIOConstants.MAX_PING_RETRIES,\n                        )\n                        raise RuntimeError(\n                            f\"Ping failed after {retry_count} retries\"\n                        ) from e\n\n                finally:\n                    time.sleep(MoRIIOConstants.PING_INTERVAL)\n                    index += 1\n\n    def shutdown(self):\n        if hasattr(self, \"moriio_wrapper\") and self.moriio_wrapper:\n            self.moriio_wrapper.shutdown()\n\n        if hasattr(self, \"_handshake_initiation_executor\"):\n            self._handshake_initiation_executor.shutdown(wait=False)\n\n        if (\n            hasattr(self, \"_moriio_handshake_listener_t\")\n            and self._moriio_handshake_listener_t\n        ):","sourceCodeStart":1413,"sourceCodeEnd":1449,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py#L1413-L1449","documentation":"The connector's keep-alive loop periodically pings the remote MoRI-IO engine over ZMQ. After MoRIIOConstants.MAX_PING_RETRIES consecutive failures (OSError or any unexpected exception per attempt) it raises RuntimeError, aborting the loop because the peer is considered unreachable.","triggerScenarios":"Sustained ping failure over MAX_PING_RETRIES intervals: peer process crashed or restarted, network partition, wrong host/port in the peer address, firewall dropping the connection.","commonSituations":"Decode instance dies while prefill keeps pinging; cross-node deployment with firewall/SecurityGroup rules blocking the ZMQ port; port offset collisions in multi-pod DP/TP layouts so pings hit nothing; NAT/DNS changes making the peer address stale.","solutions":["Check whether the peer vLLM instance is alive and restart it if it crashed","Verify network reachability of the peer host/port (telnet/nc from the pinging node) and open the ZMQ port range in the firewall","Confirm the peer address (host, base port, DP/TP port offsets) matches what the peer actually bound","If pings fail only during peer restarts, sequence restarts so producers reconnect after consumers are up"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"import socket\n\ndef peer_reachable(host: str, port: int, timeout: float = 2.0) -> bool:\n    try:\n        with socket.create_connection((host, port), timeout=timeout):\n            return True\n    except OSError:\n        return False\n\n# before enabling long-running ping loop:\nassert peer_reachable(remote_host, handshake_port), f\"peer {remote_host}:{handshake_port} unreachable\"","typeGuard":null,"tryCatchPattern":"try:\n    run_ping_loop(...)\nexcept RuntimeError as e:\n    if \"Ping failed\" in str(e):\n        # peer is down: restart/skip rather than crash the engine\n        trigger_peer_health_check_and_backoff()\n    else:\n        raise","preventionTips":["Pre-flight TCP reachability checks to peer ports before enabling the connector","Open the full ZMQ port range (base + dp/tp offsets) in firewalls between P and D nodes","Monitor peer liveness externally and stop producers when consumers die to avoid retry storms"],"tags":["network","zmq","kv-transfer","connectivity","retry"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}