{"record":{"id":"592f53260e7ef84d","repo":"sgl-project/sglang","slug":"cuda-vmm-posix-fd-broker-returned-no-file-descript","errorCode":null,"errorMessage":"CUDA VMM POSIX FD broker returned no file descriptor","messagePattern":"CUDA VMM POSIX FD broker returned no file descriptor","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/utils/cuda_vmm_transport_utils.py","lineNumber":114,"sourceCode":"    def raise_if_failed(self) -> None:\n        if self._error is not None:\n            raise RuntimeError(\"CUDA VMM POSIX FD broker failed\") from self._error\n\n    def close(self) -> None:\n        self._stop.set()\n        self._server.close()\n        self._thread.join(timeout=1.0)\n        if self._thread.is_alive():\n            raise RuntimeError(\"CUDA VMM POSIX FD broker did not stop\")\n\n\ndef _receive_posix_fd(socket_path: str) -> int:\n    with socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET) as sock:\n        sock.settimeout(_FD_SEND_TIMEOUT_S)\n        sock.connect(socket_path)\n        packet = _recv_fd(sock)\n    if packet is None:\n        raise RuntimeError(\"CUDA VMM POSIX FD broker returned no file descriptor\")\n    _src_rank, _base_idx, fd = packet\n    return fd\n\n\n@dataclass\nclass _CudaVmmMemoryChunk:\n    start: int\n    end: int\n\n    @property\n    def size(self) -> int:\n        return self.end - self.start\n\n\n@dataclass(frozen=True)\nclass _CudaVmmPackedTensorLayout:\n    relative_offset: int\n    data_nbytes: int","sourceCodeStart":96,"sourceCodeEnd":132,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/cuda_vmm_transport_utils.py#L96-L132","documentation":"A consumer connected to the POSIX FD broker socket and sent a request, but _recv_fd returned no packet — the broker did not (or could not) send an FD within the timeout before closing.","triggerScenarios":"Broker thread crashed or shut down between connect and send, send timed out (_FD_SEND_TIMEOUT_S), or ancillary FD data was dropped by the socket layer.","commonSituations":"Consumer races with pool shutdown, broker thread killed by an earlier error (see 6371), or SCM_RIGHTS truncation on unusual kernels/containers.","solutions":["Check broker-side logs for the underlying failure (often chained from 6371)","Retry once after a short delay to rule out shutdown races","Increase the FD send timeout if transfers are slow on huge pools","Avoid tearing down the pool while consumers are still importing"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"for attempt in range(2):\n    try:\n        fd = _receive_posix_fd(path)\n        break\n    except RuntimeError:\n        if attempt: raise","preventionTips":["Don't tear down the pool while consumers import","Watch broker logs for the originating failure","Tune _FD_SEND_TIMEOUT_S for large pools"],"tags":["cuda","vmm","ipc","socket","file-descriptor"],"backgroundTag":"cuda-ipc-broker-failure","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}