{"record":{"id":"a4a00f79b8728e30","repo":"sgl-project/sglang","slug":"ascend-transfer-engine-initialization-failed","errorCode":null,"errorMessage":"Ascend Transfer Engine initialization failed.","messagePattern":"Ascend Transfer Engine initialization failed\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/disaggregation/ascend/transfer_engine.py","lineNumber":84,"sourceCode":"            trans_op_type = TransferEngine.TransDataOpType.SDMA\n        else:\n            trans_op_type = TransferEngine.TransDataOpType.DEVICE_RDMA\n            \"\"\"with device RDMA for PD transfer\"\"\"\n            tmp_tensor = torch.zeros(1, device=\"npu\")\n            output_tensor_list = [\n                torch.empty_like(tmp_tensor) for _ in range(get_world_size())\n            ]\n            # Initialize hccl in advance through all_gather to avoid conflicts with rdma initialization.\n            torch.distributed.all_gather(\n                output_tensor_list, tmp_tensor, group=get_world_group().device_group\n            )\n        \"\"\"Initialize the ascend transfer instance.\"\"\"\n        ret_value = self.engine.initialize(\n            self.store_url, self.session_id, self.role, self.npu_id, trans_op_type\n        )\n        if ret_value != 0:\n            logger.error(\"Ascend Transfer Engine initialization failed.\")\n            raise RuntimeError(\"Ascend Transfer Engine initialization failed.\")\n\n    def batch_register(self, ptrs: List[int], lengths: List[int]):\n        try:\n            ret_value = self.engine.batch_register_memory(ptrs, lengths)\n        except Exception:\n            # Mark register as failed\n            ret_value = -1\n        if ret_value != 0:\n            logger.debug(f\"Ascend memory registration for ptr {ptrs} failed.\")\n\n    @staticmethod\n    def _get_transfer_protocol():\n        protocol = os.getenv(\"ASCEND_MF_TRANSFER_PROTOCOL\")\n        allowed_protocols = {\"device_rdma\", \"sdma\"}\n        if protocol and protocol.lower() in allowed_protocols:\n            return protocol.lower()\n        else:\n            logger.warning(","sourceCodeStart":66,"sourceCodeEnd":102,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/ascend/transfer_engine.py#L66-L102","documentation":"The underlying Ascend transfer engine's native initialize() returned a non-zero code, so the Python wrapper raises RuntimeError after logging. This means the C++/ACL-side session (store URL, session id, role, NPU id) could not be established.","triggerScenarios":"Calling initialize() (directly or via __init__) when the native engine fails to start: unreachable metadata store (self.store_url), duplicate session id, wrong npu_id, missing HCCL/Ascend drivers, or role string invalid.","commonSituations":"Etcd/redis metadata store for PD bootstrap not running or wrong address; NPU device id out of range for the container; Ascend CANN toolkit/driver mismatch; two processes picking the same session id (port conflict); pod started without NPU resources.","solutions":["Verify the metadata store (etcd/redis) at store_url is reachable from every PD node","Check npu_id is valid (npu-smi info) and the container has that device assigned","Check Ascend driver/CANN versions match across nodes and re-run; look at earlier native logs for the real error code","Ensure no stale session with the same session_id/hostname exists; free or change the rpc port"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"import socket\\n# verify store reachable before engine init\\nhost, port = parse(store_url)\\nsocket.create_connection((host, port), timeout=3).close()","typeGuard":null,"tryCatchPattern":"for attempt in range(3):\\n    try:\\n        engine = AscendTransferEngine(...); break\\n    except RuntimeError:\\n        if attempt == 2: raise\\n        time.sleep(2 ** attempt)","preventionTips":["Health-check the metadata store before starting PD nodes","Verify npu-smi sees the target device in each container","Pin matching CANN/driver versions across nodes"],"tags":["ascend","npu","transfer-engine","initialization","runtime"],"backgroundTag":"engine-initialization-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}