sgl-project/sglang · error · RuntimeError

POSIX fd exchange receive failed

Error message

POSIX fd exchange receive failed

What it means

After the fd exchange completes (or the receiver thread exits via exception), exchange_posix_fds re-raises the first exception captured by the receiver thread. This wraps lower-level failures such as truncated headers, duplicate fds, or send errors into a clear 'exchange failed' error.

Source

Thrown at python/sglang/srt/utils/cuda_vmm_utils.py:838

        thread = threading.Thread(target=recv_loop, daemon=True)
        thread.start()
        try:
            for peer_rank, peer_path in enumerate(paths):
                if peer_rank == rank:
                    continue
                with socket.socket(socket.AF_UNIX, sock_kind) as sock:
                    sock.settimeout(_FD_SEND_TIMEOUT_S)
                    sock.connect(peer_path)
                    for base_idx, fd in enumerate(local_fds):
                        _send_fd(sock, fd, rank, base_idx)
        finally:
            thread.join(_FD_SEND_TIMEOUT_S)

        if thread.is_alive():
            raise RuntimeError("timed out waiting for POSIX fd exchange")
        if errors:
            raise RuntimeError("POSIX fd exchange receive failed") from errors[0]

        expected = {
            (src_rank, base_idx)
            for src_rank, count in enumerate(peer_base_counts)
            if src_rank != rank
            for base_idx in range(count)
        }
        missing = expected.difference(received_fds)
        extra = set(received_fds).difference(expected)
        if missing or extra:
            for fd in received_fds.values():
                os.close(fd)
            raise RuntimeError(
                "POSIX fd exchange mismatch: "
                f"missing={sorted(missing)[:8]}, extra={sorted(extra)[:8]}"
            )
        return received_fds
    finally:

View on GitHub (pinned to 0132848349)

Solutions

1. Read the `__cause__` of this RuntimeError to identify the real underlying failure and fix that (peer crash, protocol mismatch, etc.).
2. Restart the job from a clean state after fixing the root cause.
3. Verify homogeneous sglang versions and healthy peers before rerunning the exchange.

try:
    received = exchange_posix_fds(...)
except RuntimeError as e:
    root = e.__cause__ or e
    logger.error("fd exchange failed: %s", root)
    raise

When it happens

Trigger: Any exception inside the recv_loop thread (e.g. errors 6403/6404/6405/6407) is collected into `errors`, and after the join this line raises RuntimeError chained from errors[0].

Common situations: Always accompanies one of the underlying fd-passing failures — peer crash, protocol mismatch, or duplicate sends during distributed startup.

Related errors

• duplicate fd for {key}
• timed out waiting for POSIX fd exchange
• world_size must be positive and divide global_heads
• Group {group_name} is destroyed.
• world_size ({world_size}) is less than tensor_parallel_degre

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/75814d4c085ca562. Report an issue: GitHub.