sgl-project/sglang · critical · RuntimeError

VMM handle export failed: FABRIC export failed on at least o

Error message

VMM handle export failed: FABRIC export failed on at least one rank and POSIX fd export failed on at least one rank

What it means

export_shareable_handles tries two export mechanisms for CUDA VMM handles: NVSwitch FABRIC export and POSIX fd export. If at least one rank fails FABRIC export AND at least one rank fails POSIX fd export, there is no common fallback, so it raises with the local rank's cause chained.

Source

Thrown at python/sglang/srt/utils/cuda_vmm_utils.py:772

    except Exception as e:
        posix_error = e
        posix_ok = False
        for fd in posix_fds:
            try:
                os.close(fd)
            except OSError:
                pass
        posix_fds = []

    if not all_ranks_ok(group, posix_ok):
        cause = posix_error or fabric_error
        message = (
            "VMM handle export failed: FABRIC export failed on at least one "
            "rank and POSIX fd export failed on at least one rank"
        )
        if cause is not None:
            message += f"; local rank {rank} error: {cause}"
        raise RuntimeError(message) from posix_error

    return [], posix_fds, False


def exchange_posix_fds(
    group: ProcessGroup,
    rank: int,
    world_size: int,
    local_fds: List[int],
    peer_base_counts: List[int],
):
    """Exchange POSIX file descriptors across ranks via SCM_RIGHTS over a UNIX
    socket. Returns ``{(src_rank, base_idx): fd}`` for every peer. The caller
    owns the received fds and must close them.
    """
    sock_kind = socket.SOCK_SEQPACKET
    sock_dir = tempfile.mkdtemp(prefix="sgl_ar_fd_")
    sock_path = os.path.join(sock_dir, f"rank_{rank}.sock")

View on GitHub (pinned to 0132848349)

Solutions

1. Check the chained `cause` in the traceback to see which export path failed locally and why (driver version, permission, unsupported handle type).
2. Prefer running P2P-sharing ranks on homogeneous NVSwitch nodes with a driver that supports fabric handles.
3. Ensure the container exposes /dev/* fabric device nodes and has CAP_SYS_ADMIN or appropriate permissions for fd export.

# probe capabilities before the multi-rank export
import ctypes
from python.sglang.srt.utils.cuda_vmm_utils import _get_cuda_driver
# ensure driver >= fabric-handle support and /dev devices exist before launching job

try:
    fabric_fds, posix_fds, use_fabric = export_shareable_handles(handles, group, rank)
except RuntimeError as e:
    if "VMM handle export failed" in str(e):
        # fall back to non-shared / per-rank memory strategy
        disable_peer_mapping = True
    else:
        raise

When it happens

Trigger: Calling export_shareable_handles in a multi-rank job where FABRIC (cuMemExportToShareableHandle with CU_MEM_HANDLE_TYPE_FABRIC) fails on some rank and POSIX fd export fails on another — e.g. GPUs without fabric support mixed with a container lacking /dev/fabric or fd permissions.

Common situations: Heterogeneous nodes (some with NVSwitch, some PCIe-only), older drivers lacking fabric handle support, containers missing device nodes, or IMEX channels not configured.

Related errors

• --enable-tp-lm-head-all-to-all requires an available PyNCCL
• Error happened when batch testing peer-to-peer access from {
• NCCL only supports CUDA, ROCm and MUSA backends.
• CUDA VMM multimodal transport selected POSIX_FD, but this po
• Cannot find NVIDIA Math-DX (cuBLASDx) headers. Install the `

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/4f43a78bf21a0d58. Report an issue: GitHub.