sgl-project/sglang · critical · RuntimeError

Ascend Transfer Engine initialization failed.

Error message

Ascend Transfer Engine initialization failed.

What it means

The underlying Ascend transfer engine's native initialize() returned a non-zero code, so the Python wrapper raises RuntimeError after logging. This means the C++/ACL-side session (store URL, session id, role, NPU id) could not be established.

Source

Thrown at python/sglang/srt/disaggregation/ascend/transfer_engine.py:84

            trans_op_type = TransferEngine.TransDataOpType.SDMA
        else:
            trans_op_type = TransferEngine.TransDataOpType.DEVICE_RDMA
            """with device RDMA for PD transfer"""
            tmp_tensor = torch.zeros(1, device="npu")
            output_tensor_list = [
                torch.empty_like(tmp_tensor) for _ in range(get_world_size())
            ]
            # Initialize hccl in advance through all_gather to avoid conflicts with rdma initialization.
            torch.distributed.all_gather(
                output_tensor_list, tmp_tensor, group=get_world_group().device_group
            )
        """Initialize the ascend transfer instance."""
        ret_value = self.engine.initialize(
            self.store_url, self.session_id, self.role, self.npu_id, trans_op_type
        )
        if ret_value != 0:
            logger.error("Ascend Transfer Engine initialization failed.")
            raise RuntimeError("Ascend Transfer Engine initialization failed.")

    def batch_register(self, ptrs: List[int], lengths: List[int]):
        try:
            ret_value = self.engine.batch_register_memory(ptrs, lengths)
        except Exception:
            # Mark register as failed
            ret_value = -1
        if ret_value != 0:
            logger.debug(f"Ascend memory registration for ptr {ptrs} failed.")

    @staticmethod
    def _get_transfer_protocol():
        protocol = os.getenv("ASCEND_MF_TRANSFER_PROTOCOL")
        allowed_protocols = {"device_rdma", "sdma"}
        if protocol and protocol.lower() in allowed_protocols:
            return protocol.lower()
        else:
            logger.warning(

View on GitHub (pinned to 0132848349)

Solutions

1. Verify the metadata store (etcd/redis) at store_url is reachable from every PD node
2. Check npu_id is valid (npu-smi info) and the container has that device assigned
3. Check Ascend driver/CANN versions match across nodes and re-run; look at earlier native logs for the real error code
4. Ensure no stale session with the same session_id/hostname exists; free or change the rpc port

import socket\n# verify store reachable before engine init\nhost, port = parse(store_url)\nsocket.create_connection((host, port), timeout=3).close()

for attempt in range(3):\n    try:\n        engine = AscendTransferEngine(...); break\n    except RuntimeError:\n        if attempt == 2: raise\n        time.sleep(2 ** attempt)

When it happens

Trigger: Calling initialize() (directly or via __init__) when the native engine fails to start: unreachable metadata store (self.store_url), duplicate session id, wrong npu_id, missing HCCL/Ascend drivers, or role string invalid.

Common situations: Etcd/redis metadata store for PD bootstrap not running or wrong address; NPU device id out of range for the container; Ascend CANN toolkit/driver mismatch; two processes picking the same session id (port conflict); pod started without NPU resources.

Related errors

• {name} is required for NPU packed attention
• {name} must be a 1D int32 or int64 tensor
• {name} and its host copy must have the same length
• {name} must start with 0 and contain at least one sequence
• {name} must end at the packed token count {total_tokens}, go

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a4a00f79b8728e30. Report an issue: GitHub.