{"record":{"id":"83606c95678c5faa","repo":"vllm-project/vllm","slug":"mooncake-transfer-engine-initialization-failed","errorCode":null,"errorMessage":"Mooncake Transfer Engine initialization failed.","messagePattern":"Mooncake Transfer Engine initialization failed\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py","lineNumber":941,"sourceCode":"        )\n        # Create more tasks than workers to keep the thread pool saturated.\n        # Tasks can await async events, so a surplus (2x is a robust heuristic)\n        # prevents workers from idling.\n        self.num_sender_tasks = self.num_sender_workers * 2\n        protocol = kv_transfer_config.kv_connector_extra_config.get(  # type: ignore[union-attr]\n            \"mooncake_protocol\", \"rdma\"\n        )\n        device_name = kv_transfer_config.kv_connector_extra_config.get(  # type: ignore[union-attr]\n            \"device_name\", \"\"\n        )\n        logger.info(\n            \"The Mooncake Transfer Engine is using %s as its protocol.\", protocol\n        )\n        ret_value = self.engine.initialize(\n            self.hostname, \"P2PHANDSHAKE\", protocol, device_name\n        )\n        if ret_value != 0:\n            raise RuntimeError(\"Mooncake Transfer Engine initialization failed.\")\n\n        self.rpc_port = self.engine.get_rpc_port()\n\n        logger.debug(\n            \"Mooncake Transfer Engine initialized at %s:%d\",\n            self.hostname,\n            self.rpc_port,\n        )\n\n        self._remote_agents: dict[EngineId, dict[int, dict[int, str]]] = {}\n        self._pending_bootstrap_queries: dict[str, asyncio.Event] = {}\n        self.side_channel_port: int = 0  # we will bind it in register_kv_caches()\n        self.engine_id: EngineId = engine_id\n        self.tp_rank = get_tensor_model_parallel_rank()\n        self.tp_size = get_tensor_model_parallel_world_size()\n        self.block_len_per_layer: list[int] = []\n        self.kv_block_len_per_layer: list[int] = []\n        self.registered_layer_names: list[str] = []","sourceCodeStart":923,"sourceCodeEnd":959,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py#L923-L959","documentation":"RuntimeError raised when TransferEngine.initialize(hostname, 'P2PHANDSHAKE', protocol, device_name) returns a nonzero code, meaning the Mooncake transfer engine's native initialization failed. protocol defaults to kv_connector_extra_config 'mooncake_protocol' (fallback 'rdma') and device_name likewise comes from extra config. Nonzero return codes typically mean the transport backend (RDMA/UCX verbs) could not be set up on this host — missing drivers, no RDMA devices, or an unsupported protocol/device string.","triggerScenarios":"protocol 'rdma' on a host without RDMA NICs or ibverbs installed; wrong device_name in kv_connector_extra_config; hostname not resolvable/bindable; ucx protocol requested without UCX installed; running in a plain container without /dev/infiniband devices.","commonSituations":"Testing the Mooncake connector on non-RDMA hardware; Kubernetes pods missing InfiniBand device mounts; typo'd mooncake_protocol or device_name in kv_connector_extra_config; mlx/p2p protocol attempted on unsupported hardware.","solutions":["Check the configured mooncake_protocol and device_name in kv_connector_extra_config and confirm the host supports that transport.","On hosts without RDMA hardware, use a software transport supported by mooncake (e.g. set protocol to 'ucx' with tcp/posix transport or per mooncake docs) or enable the appropriate mode.","Verify RDMA stack health: ibv_devinfo lists devices, ibverbs/ucx libraries installed.","In containers, ensure InfiniBand devices and libraries are mounted/installed."],"exampleFix":"# before\nextra_config = {\"mooncake_protocol\": \"rdma\"}  # host has no RDMA NIC\n# after\nextra_config = {\"mooncake_protocol\": \"ucx\"}  # UCX over TCP/POSIX","handlingStrategy":"fallback","validationCode":"import subprocess\ndef rdma_devices_present() -> bool:\n    return subprocess.run([\"ibv_devinfo\"], capture_output=True).returncode == 0\n# choose protocol = 'rdma' only if rdma_devices_present() else 'ucx'","typeGuard":null,"tryCatchPattern":"Catch RuntimeError from engine init; inspect protocol/device_name config, switch the transport (e.g. rdma -> ucx) and retry initialization once with the corrected config.","preventionTips":["Probe host RDMA capability during deployment planning","Validate mooncake_protocol and device_name against the hardware before engine start","Keep RDMA drivers and ibverbs/ucx libraries pinned in the container image"],"tags":["mooncake","rdma","initialization","config","kv-transfer"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}