{"record":{"id":"4dd5642bb63135b4","repo":"sgl-project/sglang","slug":"cuda-vmm-posix-fd-broker-failed","errorCode":null,"errorMessage":"CUDA VMM POSIX FD broker failed","messagePattern":"CUDA VMM POSIX FD broker failed","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/utils/cuda_vmm_transport_utils.py","lineNumber":98,"sourceCode":"                continue\n            except OSError as error:\n                if self._stop.is_set():\n                    return\n                self._error = error\n                logger.exception(\"CUDA VMM POSIX FD broker failed\")\n                return\n\n            try:\n                with conn:\n                    _send_fd(conn, self.fd, src_rank=0, base_idx=0)\n            except Exception as error:\n                self._error = error\n                logger.exception(\"CUDA VMM POSIX FD broker failed\")\n                return\n\n    def raise_if_failed(self) -> None:\n        if self._error is not None:\n            raise RuntimeError(\"CUDA VMM POSIX FD broker failed\") from self._error\n\n    def close(self) -> None:\n        self._stop.set()\n        self._server.close()\n        self._thread.join(timeout=1.0)\n        if self._thread.is_alive():\n            raise RuntimeError(\"CUDA VMM POSIX FD broker did not stop\")\n\n\ndef _receive_posix_fd(socket_path: str) -> int:\n    with socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET) as sock:\n        sock.settimeout(_FD_SEND_TIMEOUT_S)\n        sock.connect(socket_path)\n        packet = _recv_fd(sock)\n    if packet is None:\n        raise RuntimeError(\"CUDA VMM POSIX FD broker returned no file descriptor\")\n    _src_rank, _base_idx, fd = packet\n    return fd","sourceCodeStart":80,"sourceCodeEnd":116,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/cuda_vmm_transport_utils.py#L80-L116","documentation":"The CUDA VMM POSIX FD broker thread hit an exception while serving file descriptors; it records the error and raise_if_failed re-raises it as this RuntimeError at the next pool interaction. The original exception is chained via 'from self._error'.","triggerScenarios":"The broker's Unix-socket server failing (socket bind/accept errors, CUDA driver errors while exporting FDs); the failure surfaces later when the pool calls _raise_if_failed.","commonSituations":"Multimodal KV-transfer VMM pools on machines where POSIX FD handle export is unsupported or the socket path is inaccessible; transient driver faults.","solutions":["Read the chained __cause__ — it holds the real broker exception","Check that the GPU/driver supports CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR (nvidia-smi, driver >= the VMM requirement)","Verify socket path permissions/tmpfs availability","Retry with FABRIC handle type hardware or disable the VMM transport fallback if unsupported"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    pool.wrap_tensors(ts)\nexcept RuntimeError as e:\n    if \"broker failed\" in str(e) and e.__cause__:\n        diagnose(e.__cause__)","preventionTips":["Verify POSIX FD handle support and driver version before enabling the VMM transport","Log broker exceptions when they first occur (they surface lazily)","Avoid mixing fabric/POSIX pools on unsupported topologies"],"tags":["cuda","vmm","ipc","file-descriptor","broker"],"backgroundTag":"cuda-ipc-broker-failure","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}