sgl-project/sglang · error · RuntimeError

timed out waiting for POSIX fd exchange

Error message

timed out waiting for POSIX fd exchange

What it means

exchange_posix_fds runs a receiver thread and joins it with _FD_SEND_TIMEOUT_S; if the thread is still alive after the timeout (a peer never connected or never finished sending), the exchange is declared timed out.

Source

Thrown at python/sglang/srt/utils/cuda_vmm_utils.py:836

        paths = [None] * world_size
        dist.all_gather_object(paths, sock_path, group=group)

        thread = threading.Thread(target=recv_loop, daemon=True)
        thread.start()
        try:
            for peer_rank, peer_path in enumerate(paths):
                if peer_rank == rank:
                    continue
                with socket.socket(socket.AF_UNIX, sock_kind) as sock:
                    sock.settimeout(_FD_SEND_TIMEOUT_S)
                    sock.connect(peer_path)
                    for base_idx, fd in enumerate(local_fds):
                        _send_fd(sock, fd, rank, base_idx)
        finally:
            thread.join(_FD_SEND_TIMEOUT_S)

        if thread.is_alive():
            raise RuntimeError("timed out waiting for POSIX fd exchange")
        if errors:
            raise RuntimeError("POSIX fd exchange receive failed") from errors[0]

        expected = {
            (src_rank, base_idx)
            for src_rank, count in enumerate(peer_base_counts)
            if src_rank != rank
            for base_idx in range(count)
        }
        missing = expected.difference(received_fds)
        extra = set(received_fds).difference(expected)
        if missing or extra:
            for fd in received_fds.values():
                os.close(fd)
            raise RuntimeError(
                "POSIX fd exchange mismatch: "
                f"missing={sorted(missing)[:8]}, extra={sorted(extra)[:8]}"
            )

View on GitHub (pinned to 0132848349)

Solutions

1. Inspect other ranks' logs for a crashed/hung peer — usually the root cause.
2. Ensure all ranks use the same socket path/tmpdir derivation.
3. Increase _FD_SEND_TIMEOUT_S if the cluster is slow or world size is large.

try:
    fds = exchange_posix_fds(group, rank, local_fds, sock_path, peer_counts)
except RuntimeError as e:
    if "timed out" in str(e):
        check_peer_ranks_alive()  # usually a crashed peer
        raise

When it happens

Trigger: The recv thread does not finish within _FD_SEND_TIMEOUT_S seconds — a peer rank crashed before connecting, socket path mismatch, or a sender blocked on sendmsg because the peer isn't draining.

Common situations: One rank dies during startup (OOM, CUDA init failure) so it never sends its fds; mismatched socket paths from differing tmp dirs; very slow/large-rank jobs exceeding the default timeout.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• sendmsg sent {sent} bytes, expected {len(header)}
• received truncated fd header: {len(data)} < {_FD_HEADER_BYTE
• duplicate fd for {key}
• POSIX fd exchange receive failed
• Waiting for main node timeout!

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/54b03021d6f37a66. Report an issue: GitHub.