sgl-project/sglang · critical · RuntimeError

CUDA VMM POSIX FD broker returned no file descriptor

Error message

CUDA VMM POSIX FD broker returned no file descriptor

What it means

A consumer connected to the POSIX FD broker socket and sent a request, but _recv_fd returned no packet — the broker did not (or could not) send an FD within the timeout before closing.

Source

Thrown at python/sglang/srt/utils/cuda_vmm_transport_utils.py:114

    def raise_if_failed(self) -> None:
        if self._error is not None:
            raise RuntimeError("CUDA VMM POSIX FD broker failed") from self._error

    def close(self) -> None:
        self._stop.set()
        self._server.close()
        self._thread.join(timeout=1.0)
        if self._thread.is_alive():
            raise RuntimeError("CUDA VMM POSIX FD broker did not stop")


def _receive_posix_fd(socket_path: str) -> int:
    with socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET) as sock:
        sock.settimeout(_FD_SEND_TIMEOUT_S)
        sock.connect(socket_path)
        packet = _recv_fd(sock)
    if packet is None:
        raise RuntimeError("CUDA VMM POSIX FD broker returned no file descriptor")
    _src_rank, _base_idx, fd = packet
    return fd


@dataclass
class _CudaVmmMemoryChunk:
    start: int
    end: int

    @property
    def size(self) -> int:
        return self.end - self.start


@dataclass(frozen=True)
class _CudaVmmPackedTensorLayout:
    relative_offset: int
    data_nbytes: int

View on GitHub (pinned to 0132848349)

Solutions

1. Check broker-side logs for the underlying failure (often chained from 6371)
2. Retry once after a short delay to rule out shutdown races
3. Increase the FD send timeout if transfers are slow on huge pools
4. Avoid tearing down the pool while consumers are still importing

for attempt in range(2):
    try:
        fd = _receive_posix_fd(path)
        break
    except RuntimeError:
        if attempt: raise

When it happens

Trigger: Broker thread crashed or shut down between connect and send, send timed out (_FD_SEND_TIMEOUT_S), or ancillary FD data was dropped by the socket layer.

Common situations: Consumer races with pool shutdown, broker thread killed by an earlier error (see 6371), or SCM_RIGHTS truncation on unusual kernels/containers.

Related errors

• CUDA VMM POSIX FD broker failed
• CUDA VMM proxy has no shareable handle
• --mm-feature-transport=cuda_ipc requires NVIDIA CUDA.
• memory_size must be positive
• consumer_count must be positive

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/592f53260e7ef84d. Report an issue: GitHub.