{"record":{"id":"54b03021d6f37a66","repo":"sgl-project/sglang","slug":"timed-out-waiting-for-posix-fd-exchange","errorCode":null,"errorMessage":"timed out waiting for POSIX fd exchange","messagePattern":"timed out waiting for POSIX fd exchange","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/utils/cuda_vmm_utils.py","lineNumber":836,"sourceCode":"        paths = [None] * world_size\n        dist.all_gather_object(paths, sock_path, group=group)\n\n        thread = threading.Thread(target=recv_loop, daemon=True)\n        thread.start()\n        try:\n            for peer_rank, peer_path in enumerate(paths):\n                if peer_rank == rank:\n                    continue\n                with socket.socket(socket.AF_UNIX, sock_kind) as sock:\n                    sock.settimeout(_FD_SEND_TIMEOUT_S)\n                    sock.connect(peer_path)\n                    for base_idx, fd in enumerate(local_fds):\n                        _send_fd(sock, fd, rank, base_idx)\n        finally:\n            thread.join(_FD_SEND_TIMEOUT_S)\n\n        if thread.is_alive():\n            raise RuntimeError(\"timed out waiting for POSIX fd exchange\")\n        if errors:\n            raise RuntimeError(\"POSIX fd exchange receive failed\") from errors[0]\n\n        expected = {\n            (src_rank, base_idx)\n            for src_rank, count in enumerate(peer_base_counts)\n            if src_rank != rank\n            for base_idx in range(count)\n        }\n        missing = expected.difference(received_fds)\n        extra = set(received_fds).difference(expected)\n        if missing or extra:\n            for fd in received_fds.values():\n                os.close(fd)\n            raise RuntimeError(\n                \"POSIX fd exchange mismatch: \"\n                f\"missing={sorted(missing)[:8]}, extra={sorted(extra)[:8]}\"\n            )","sourceCodeStart":818,"sourceCodeEnd":854,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/cuda_vmm_utils.py#L818-L854","documentation":"exchange_posix_fds runs a receiver thread and joins it with _FD_SEND_TIMEOUT_S; if the thread is still alive after the timeout (a peer never connected or never finished sending), the exchange is declared timed out.","triggerScenarios":"The recv thread does not finish within _FD_SEND_TIMEOUT_S seconds — a peer rank crashed before connecting, socket path mismatch, or a sender blocked on sendmsg because the peer isn't draining.","commonSituations":"One rank dies during startup (OOM, CUDA init failure) so it never sends its fds; mismatched socket paths from differing tmp dirs; very slow/large-rank jobs exceeding the default timeout.","solutions":["Inspect other ranks' logs for a crashed/hung peer — usually the root cause.","Ensure all ranks use the same socket path/tmpdir derivation.","Increase _FD_SEND_TIMEOUT_S if the cluster is slow or world size is large."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    fds = exchange_posix_fds(group, rank, local_fds, sock_path, peer_counts)\nexcept RuntimeError as e:\n    if \"timed out\" in str(e):\n        check_peer_ranks_alive()  # usually a crashed peer\n        raise","preventionTips":["Log peer health before exchange","Derive socket path identically on all ranks","Raise _FD_SEND_TIMEOUT_S for large world sizes"],"tags":["distributed","timeout","unix-socket","fd-passing"],"backgroundTag":"operation-timed-out","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}