sgl-project/sglang · critical · RuntimeError

Mooncake Transfer Engine initialization failed.

Error message

Mooncake Transfer Engine initialization failed.

What it means

MooncakeTransferEngine.initialize calls the native engine's initialize (with metadata, protocol P2PHANDSHAKE, device name) and checks the return code; a nonzero return means the underlying mooncake transfer engine failed to initialize (e.g. RDMA device issue, bad metadata/protocol, IPC failure). SGLang surfaces this as a generic RuntimeError after logging the same message.

Source

Thrown at python/sglang/srt/distributed/device_communicators/mooncake_transfer_engine.py:218

        if envs.ENABLE_ASCEND_TRANSFER_WITH_MOONCAKE.get():
            npu_phy_id = envs.ASCEND_NPU_PHY_ID.get()
            suffix = self.gpu_id if npu_phy_id == -1 else npu_phy_id
            hostname += f":{get_free_port()}:npu_{suffix}"
            protocol = "ascend"
        else:
            # MOONCAKE_PROTOCOL selects the transport (rdma | efa | tcp | ...).
            # Default is "rdma"; set MOONCAKE_PROTOCOL=efa on AWS EFA hardware.
            protocol = envs.MOONCAKE_PROTOCOL.get()

        ret_value = self.engine.initialize(
            hostname,
            "P2PHANDSHAKE",
            protocol,
            device_name if device_name is not None else "",
        )
        if ret_value != 0:
            logger.error("Mooncake Transfer Engine initialization failed.")
            raise RuntimeError("Mooncake Transfer Engine initialization failed.")

    def transfer_sync(
        self, session_id: str, buffer: int, peer_buffer_address: int, length: int
    ) -> int:
        """Synchronously transfer data to the specified address."""
        try:
            ret = self.engine.transfer_sync_write(
                session_id, buffer, peer_buffer_address, length
            )
        except Exception:
            ret = -1

        if ret < 0:
            logger.debug(
                "Failed to transfer data from %s to %s - %s.",
                buffer,
                session_id,
                peer_buffer_address,

View on GitHub (pinned to 0132848349)

Solutions

1. Check the logger.error output immediately preceding the raise — the native return code and mooncake logs identify the failing step
2. Verify RDMA stack: ibv_devinfo should list devices; ensure drivers and ibverbs are installed and the container has RDMA device access
3. Confirm the protocol and device_name passed to initialize are valid for your host (correct NIC/interface name)
4. Restart/check mooncake-dependent services (metadata store) and retry; if on a single box without RDMA, use a supported transport or disable mooncake

import subprocess
# sanity: RDMA devices present before mooncake init
subprocess.run(['ibv_devinfo'], check=True, capture_output=True)

try:
    mc.initialize(...)
except RuntimeError as e:
    if 'initialization failed' in str(e):
        logger.error('mooncake init failed; check RDMA/device config: %s', e)
        # degrade to non-mooncake transfer or fail fast with diagnostics
    raise

When it happens

Trigger: Calling initialize() on MooncakeTransferEngine (from __init__ or explicit init) where engine.initialize(...) returns non-zero — typically wrong/absent RDMA or NVLink device, unsupported protocol string, device_name resolution failure, or daemon/transport problems in mooncake.

Common situations: Running mooncake-based transfer on nodes without proper RDMA NICs or drivers (missing ibverbs); passing a device name that doesn't exist; mooncake metadata server/daemon not running; container without RDMA devices mounted or /dev/shm misconfigured; firewall blocking P2P handshake ports.

Related errors

• mooncake encoder_transfer_backend requires HTTP encoders; us
• Mooncake transfer_sync failed for {req_id} (session={session
• [Staging] Bulk RDMA transfer failed with ret={ret}. src_ptr=
• SGLANG_DISAGG_STAGING_BUFFER with pp_size > 1 is only suppor
• Please install mooncake by following the instructions at htt

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7cb35a3376a5f01c. Report an issue: GitHub.