{"record":{"id":"7cb35a3376a5f01c","repo":"sgl-project/sglang","slug":"mooncake-transfer-engine-initialization-failed","errorCode":null,"errorMessage":"Mooncake Transfer Engine initialization failed.","messagePattern":"Mooncake Transfer Engine initialization failed\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/distributed/device_communicators/mooncake_transfer_engine.py","lineNumber":218,"sourceCode":"        if envs.ENABLE_ASCEND_TRANSFER_WITH_MOONCAKE.get():\n            npu_phy_id = envs.ASCEND_NPU_PHY_ID.get()\n            suffix = self.gpu_id if npu_phy_id == -1 else npu_phy_id\n            hostname += f\":{get_free_port()}:npu_{suffix}\"\n            protocol = \"ascend\"\n        else:\n            # MOONCAKE_PROTOCOL selects the transport (rdma | efa | tcp | ...).\n            # Default is \"rdma\"; set MOONCAKE_PROTOCOL=efa on AWS EFA hardware.\n            protocol = envs.MOONCAKE_PROTOCOL.get()\n\n        ret_value = self.engine.initialize(\n            hostname,\n            \"P2PHANDSHAKE\",\n            protocol,\n            device_name if device_name is not None else \"\",\n        )\n        if ret_value != 0:\n            logger.error(\"Mooncake Transfer Engine initialization failed.\")\n            raise RuntimeError(\"Mooncake Transfer Engine initialization failed.\")\n\n    def transfer_sync(\n        self, session_id: str, buffer: int, peer_buffer_address: int, length: int\n    ) -> int:\n        \"\"\"Synchronously transfer data to the specified address.\"\"\"\n        try:\n            ret = self.engine.transfer_sync_write(\n                session_id, buffer, peer_buffer_address, length\n            )\n        except Exception:\n            ret = -1\n\n        if ret < 0:\n            logger.debug(\n                \"Failed to transfer data from %s to %s - %s.\",\n                buffer,\n                session_id,\n                peer_buffer_address,","sourceCodeStart":200,"sourceCodeEnd":236,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/distributed/device_communicators/mooncake_transfer_engine.py#L200-L236","documentation":"MooncakeTransferEngine.initialize calls the native engine's initialize (with metadata, protocol P2PHANDSHAKE, device name) and checks the return code; a nonzero return means the underlying mooncake transfer engine failed to initialize (e.g. RDMA device issue, bad metadata/protocol, IPC failure). SGLang surfaces this as a generic RuntimeError after logging the same message.","triggerScenarios":"Calling initialize() on MooncakeTransferEngine (from __init__ or explicit init) where engine.initialize(...) returns non-zero — typically wrong/absent RDMA or NVLink device, unsupported protocol string, device_name resolution failure, or daemon/transport problems in mooncake.","commonSituations":"Running mooncake-based transfer on nodes without proper RDMA NICs or drivers (missing ibverbs); passing a device name that doesn't exist; mooncake metadata server/daemon not running; container without RDMA devices mounted or /dev/shm misconfigured; firewall blocking P2P handshake ports.","solutions":["Check the logger.error output immediately preceding the raise — the native return code and mooncake logs identify the failing step","Verify RDMA stack: ibv_devinfo should list devices; ensure drivers and ibverbs are installed and the container has RDMA device access","Confirm the protocol and device_name passed to initialize are valid for your host (correct NIC/interface name)","Restart/check mooncake-dependent services (metadata store) and retry; if on a single box without RDMA, use a supported transport or disable mooncake"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"import subprocess\n# sanity: RDMA devices present before mooncake init\nsubprocess.run(['ibv_devinfo'], check=True, capture_output=True)","typeGuard":null,"tryCatchPattern":"try:\n    mc.initialize(...)\nexcept RuntimeError as e:\n    if 'initialization failed' in str(e):\n        logger.error('mooncake init failed; check RDMA/device config: %s', e)\n        # degrade to non-mooncake transfer or fail fast with diagnostics\n    raise","preventionTips":["Validate ibv_devinfo lists devices before enabling mooncake","Mount RDMA devices and sufficient /dev/shm in containers","Smoke-test mooncake initialize on each node before full serving"],"tags":["mooncake","rdma","initialization","native-return-code","sglang"],"backgroundTag":"rdma-initialization-failure","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}