{"record":{"id":"75814d4c085ca562","repo":"sgl-project/sglang","slug":"posix-fd-exchange-receive-failed","errorCode":null,"errorMessage":"POSIX fd exchange receive failed","messagePattern":"POSIX fd exchange receive failed","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/utils/cuda_vmm_utils.py","lineNumber":838,"sourceCode":"\n        thread = threading.Thread(target=recv_loop, daemon=True)\n        thread.start()\n        try:\n            for peer_rank, peer_path in enumerate(paths):\n                if peer_rank == rank:\n                    continue\n                with socket.socket(socket.AF_UNIX, sock_kind) as sock:\n                    sock.settimeout(_FD_SEND_TIMEOUT_S)\n                    sock.connect(peer_path)\n                    for base_idx, fd in enumerate(local_fds):\n                        _send_fd(sock, fd, rank, base_idx)\n        finally:\n            thread.join(_FD_SEND_TIMEOUT_S)\n\n        if thread.is_alive():\n            raise RuntimeError(\"timed out waiting for POSIX fd exchange\")\n        if errors:\n            raise RuntimeError(\"POSIX fd exchange receive failed\") from errors[0]\n\n        expected = {\n            (src_rank, base_idx)\n            for src_rank, count in enumerate(peer_base_counts)\n            if src_rank != rank\n            for base_idx in range(count)\n        }\n        missing = expected.difference(received_fds)\n        extra = set(received_fds).difference(expected)\n        if missing or extra:\n            for fd in received_fds.values():\n                os.close(fd)\n            raise RuntimeError(\n                \"POSIX fd exchange mismatch: \"\n                f\"missing={sorted(missing)[:8]}, extra={sorted(extra)[:8]}\"\n            )\n        return received_fds\n    finally:","sourceCodeStart":820,"sourceCodeEnd":856,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/cuda_vmm_utils.py#L820-L856","documentation":"After the fd exchange completes (or the receiver thread exits via exception), exchange_posix_fds re-raises the first exception captured by the receiver thread. This wraps lower-level failures such as truncated headers, duplicate fds, or send errors into a clear 'exchange failed' error.","triggerScenarios":"Any exception inside the recv_loop thread (e.g. errors 6403/6404/6405/6407) is collected into `errors`, and after the join this line raises RuntimeError chained from errors[0].","commonSituations":"Always accompanies one of the underlying fd-passing failures — peer crash, protocol mismatch, or duplicate sends during distributed startup.","solutions":["Read the `__cause__` of this RuntimeError to identify the real underlying failure and fix that (peer crash, protocol mismatch, etc.).","Restart the job from a clean state after fixing the root cause.","Verify homogeneous sglang versions and healthy peers before rerunning the exchange."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    received = exchange_posix_fds(...)\nexcept RuntimeError as e:\n    root = e.__cause__ or e\n    logger.error(\"fd exchange failed: %s\", root)\n    raise","preventionTips":["Always inspect __cause__ to find the real failure","Restart the whole job, not individual ranks, after fd-exchange failures"],"tags":["distributed","fd-passing","wrapper-error"],"backgroundTag":"distributed-exchange-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}