sgl-project/sglang · critical · RuntimeError

CUDA VMM POSIX FD broker returned no file descriptor

Error message

CUDA VMM POSIX FD broker returned no file descriptor

What it means

CUDA VMM (virtual memory management) FD transport error: the broker process responsible for passing POSIX file descriptors returned no FD for the requested share. Usually means the broker died, the peer closed early, or the shared handle was never registered.

Solutions

1. Check broker-side logs for the underlying failure (often chained from 6371)
2. Retry once after a short delay to rule out shutdown races
3. Increase the FD send timeout if transfers are slow on huge pools
4. Avoid tearing down the pool while consumers are still importing

for attempt in range(2):
    try:
        fd = _receive_posix_fd(path)
        break
    except RuntimeError:
        if attempt: raise

When it happens

Trigger: Broker thread crashed or shut down between connect and send, send timed out (_FD_SEND_TIMEOUT_S), or ancillary FD data was dropped by the socket layer.

Common situations: Consumer races with pool shutdown, broker thread killed by an earlier error (see 6371), or SCM_RIGHTS truncation on unusual kernels/containers.

Related errors

• CUDA VMM POSIX FD broker failed
• CUDA VMM proxy has no shareable handle
• A CUDA VMM-enabled model must provide a multimodal processor
• attention group range
• consumer_count must be 1, the attention TP size, or the…

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/592f53260e7ef84d. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/srt/utils/cuda_vmm_transport_utils.py:114

    def raise_if_failed(self) -> None:
        if self._error is not None:
            raise RuntimeError("CUDA VMM POSIX FD broker failed") from self._error

    def close(self) -> None:
        self._stop.set()
        self._server.close()
        self._thread.join(timeout=1.0)
        if self._thread.is_alive():
            raise RuntimeError("CUDA VMM POSIX FD broker did not stop")


def _receive_posix_fd(socket_path: str) -> int:
    with socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET) as sock:
        sock.settimeout(_FD_SEND_TIMEOUT_S)
        sock.connect(socket_path)
        packet = _recv_fd(sock)
    if packet is None:
        raise RuntimeError("CUDA VMM POSIX FD broker returned no file descriptor")
    _src_rank, _base_idx, fd = packet
    return fd


@dataclass
class _CudaVmmMemoryChunk:
    start: int
    end: int

    @property
    def size(self) -> int:
        return self.end - self.start


@dataclass(frozen=True)
class _CudaVmmPackedTensorLayout:
    relative_offset: int
    data_nbytes: int

View on GitHub (pinned to 0132848349)