sgl-project/sglang · critical · RuntimeError

CUDA VMM POSIX FD broker failed

Error message

CUDA VMM POSIX FD broker failed

What it means

The CUDA VMM POSIX FD broker thread hit an exception while serving file descriptors; it records the error and raise_if_failed re-raises it as this RuntimeError at the next pool interaction. The original exception is chained via 'from self._error'.

Source

Thrown at python/sglang/srt/utils/cuda_vmm_transport_utils.py:98

                continue
            except OSError as error:
                if self._stop.is_set():
                    return
                self._error = error
                logger.exception("CUDA VMM POSIX FD broker failed")
                return

            try:
                with conn:
                    _send_fd(conn, self.fd, src_rank=0, base_idx=0)
            except Exception as error:
                self._error = error
                logger.exception("CUDA VMM POSIX FD broker failed")
                return

    def raise_if_failed(self) -> None:
        if self._error is not None:
            raise RuntimeError("CUDA VMM POSIX FD broker failed") from self._error

    def close(self) -> None:
        self._stop.set()
        self._server.close()
        self._thread.join(timeout=1.0)
        if self._thread.is_alive():
            raise RuntimeError("CUDA VMM POSIX FD broker did not stop")


def _receive_posix_fd(socket_path: str) -> int:
    with socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET) as sock:
        sock.settimeout(_FD_SEND_TIMEOUT_S)
        sock.connect(socket_path)
        packet = _recv_fd(sock)
    if packet is None:
        raise RuntimeError("CUDA VMM POSIX FD broker returned no file descriptor")
    _src_rank, _base_idx, fd = packet
    return fd

View on GitHub (pinned to 0132848349)

Solutions

1. Read the chained __cause__ — it holds the real broker exception
2. Check that the GPU/driver supports CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR (nvidia-smi, driver >= the VMM requirement)
3. Verify socket path permissions/tmpfs availability
4. Retry with FABRIC handle type hardware or disable the VMM transport fallback if unsupported

try:
    pool.wrap_tensors(ts)
except RuntimeError as e:
    if "broker failed" in str(e) and e.__cause__:
        diagnose(e.__cause__)

When it happens

Trigger: The broker's Unix-socket server failing (socket bind/accept errors, CUDA driver errors while exporting FDs); the failure surfaces later when the pool calls _raise_if_failed.

Common situations: Multimodal KV-transfer VMM pools on machines where POSIX FD handle export is unsupported or the socket path is inaccessible; transient driver faults.

Related errors

• CUDA VMM POSIX FD broker returned no file descriptor
• CUDA VMM proxy has no shareable handle
• --mm-feature-transport=cuda_ipc requires NVIDIA CUDA.
• memory_size must be positive
• consumer_count must be positive

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/4dd5642bb63135b4. Report an issue: GitHub.