sgl-project/sglang · critical · RuntimeError

CUDA VMM multimodal transport selected POSIX_FD, but this po

Error message

CUDA VMM multimodal transport selected POSIX_FD, but this pool requires FABRIC

What it means

The device only supports the POSIX FD allocation handle type, but this VMM pool requires FABRIC handles (GPUDirect/fabric-attached memory) and allow_posix_fallback is False. The pool refuses to start rather than silently degrading.

Source

Thrown at python/sglang/srt/utils/cuda_vmm_transport_utils.py:217

        self._closed = False

        self._allocation: VmmReservation | None = None
        self.allocation_size = 0
        self.shareable_handle = None
        self.memory_pool = None
        self._fd_broker: _PosixFdBroker | None = None
        self.posix_socket_path: str | None = None
        self._recycle_stream = None
        self._recycle_thread = None

        drv = _get_cuda_driver()
        fabric = drv.CUmemAllocationHandleType.CU_MEM_HANDLE_TYPE_FABRIC
        posix_fd = (
            drv.CUmemAllocationHandleType.CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR
        )
        self.handle_type = get_device_allocation_handle_type(self.device_index)
        if self.handle_type == posix_fd and not allow_posix_fallback:
            raise RuntimeError(
                "CUDA VMM multimodal transport selected POSIX_FD, but this "
                "pool requires FABRIC"
            )
        self.use_fabric = self.handle_type == fabric
        try:
            self._allocate(memory_size)
        except RuntimeError as error:
            if not allow_posix_fallback or self.handle_type != fabric:
                raise
            logger.warning(
                "CUDA FABRIC VMM allocation is unavailable; falling back to "
                "a POSIX FD handle: %s",
                error,
            )
            self.handle_type = posix_fd
            self.use_fabric = False
            self._allocate(memory_size)
        try:

View on GitHub (pinned to 0132848349)

Solutions

1. Enable allow_posix_fallback if the topology is safe for FD-based sharing
2. Use hardware/driver that supports fabric handles for this pool
3. Or select a different multimodal transport that does not require FABRIC

Example fix

# before
pool = CudaVmmTransportPool(..., allow_posix_fallback=False)
# after
pool = CudaVmmTransportPool(..., allow_posix_fallback=True)  # single-node / FD sharing OK

from cuda import cuda as drv
# probe handle support before constructing the pool
handle = get_device_allocation_handle_type(gpu_id)

try:
    pool = Pool(..., allow_posix_fallback=False)
except RuntimeError as e:
    if "requires FABRIC" in str(e):
        pool = Pool(..., allow_posix_fallback=True)

When it happens

Trigger: Running the multimodal VMM transport on hardware/driver combos where CU_MEM_HANDLE_TYPE_FABRIC is unavailable while the code path demands FABRIC interconnect (multi-node NVLink/fabric setups), with the fallback flag disabled.

Common situations: Single-node non-fabric GPUs, older drivers without fabric handle support, or a new deployment copied from a fabric cluster config.

Related errors

• CUDA VMM multimodal pool failed
• CUDA VMM POSIX FD broker failed
• CUDA VMM POSIX FD broker returned no file descriptor
• memory_size must be positive
• consumer_count must be positive

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/df4829303da5f5bc. Report an issue: GitHub.