{"record":{"id":"592f53260e7ef84d","repo":"sgl-project/sglang","slug":"cuda-vmm-posix-fd-broker-returned-no-file-descript","errorCode":null,"errorMessage":"CUDA VMM POSIX FD broker returned no file descriptor","messagePattern":"CUDA VMM POSIX FD broker returned no file descriptor","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/utils/cuda_vmm_transport_utils.py","lineNumber":114,"sourceCode":"    def raise_if_failed(self) -> None:\n        if self._error is not None:\n            raise RuntimeError(\"CUDA VMM POSIX FD broker failed\") from self._error\n\n    def close(self) -> None:\n        self._stop.set()\n        self._server.close()\n        self._thread.join(timeout=1.0)\n        if self._thread.is_alive():\n            raise RuntimeError(\"CUDA VMM POSIX FD broker did not stop\")\n\n\ndef _receive_posix_fd(socket_path: str) -> int:\n    with socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET) as sock:\n        sock.settimeout(_FD_SEND_TIMEOUT_S)\n        sock.connect(socket_path)\n        packet = _recv_fd(sock)\n    if packet is None:\n        raise RuntimeError(\"CUDA VMM POSIX FD broker returned no file descriptor\")\n    _src_rank, _base_idx, fd = packet\n    return fd\n\n\n@dataclass\nclass _CudaVmmMemoryChunk:\n    start: int\n    end: int\n\n    @property\n    def size(self) -> int:\n        return self.end - self.start\n\n\n@dataclass(frozen=True)\nclass _CudaVmmPackedTensorLayout:\n    relative_offset: int\n    data_nbytes: int","sourceCodeStart":96,"sourceCodeEnd":132,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/cuda_vmm_transport_utils.py#L96-L132","documentation":"CUDA VMM (virtual memory management) FD transport error: the broker process responsible for passing POSIX file descriptors returned no FD for the requested share. Usually means the broker died, the peer closed early, or the shared handle was never registered.","triggerScenarios":"Broker thread crashed or shut down between connect and send, send timed out (_FD_SEND_TIMEOUT_S), or ancillary FD data was dropped by the socket layer.","commonSituations":"Consumer races with pool shutdown, broker thread killed by an earlier error (see 6371), or SCM_RIGHTS truncation on unusual kernels/containers.","solutions":["Check broker-side logs for the underlying failure (often chained from 6371)","Retry once after a short delay to rule out shutdown races","Increase the FD send timeout if transfers are slow on huge pools","Avoid tearing down the pool while consumers are still importing"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"for attempt in range(2):\n    try:\n        fd = _receive_posix_fd(path)\n        break\n    except RuntimeError:\n        if attempt: raise","preventionTips":["Don't tear down the pool while consumers import","Watch broker logs for the originating failure","Tune _FD_SEND_TIMEOUT_S for large pools"],"tags":["cuda","vmm","ipc","socket","file-descriptor"],"backgroundTag":"cuda-ipc-broker-failure","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","contentChangedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}