sgl-project/sglang · error · RuntimeError

duplicate fd for {key}

Error message

duplicate fd for {key}

What it means

During the POSIX fd exchange receive loop, a duplicate (src_rank, base_idx) key means the same peer segment was delivered twice. The fd is closed and an error raised to avoid double-mapping or leaking a handle, indicating a misbehaving sender or protocol violation.

Source

Thrown at python/sglang/srt/utils/cuda_vmm_utils.py:810

    server.settimeout(_FD_SEND_TIMEOUT_S)
    received_fds = {}
    errors = []

    def recv_loop():
        try:
            for _ in range(world_size - 1):
                conn, _ = server.accept()
                with conn:
                    conn.settimeout(_FD_SEND_TIMEOUT_S)
                    while True:
                        packet = _recv_fd(conn)
                        if packet is None:
                            break
                        src_rank, base_idx, fd = packet
                        key = (src_rank, base_idx)
                        if key in received_fds:
                            os.close(fd)
                            raise RuntimeError(f"duplicate fd for {key}")
                        received_fds[key] = fd
        except BaseException as e:
            errors.append(e)

    try:
        server.bind(sock_path)
        server.listen(world_size)
        paths = [None] * world_size
        dist.all_gather_object(paths, sock_path, group=group)

        thread = threading.Thread(target=recv_loop, daemon=True)
        thread.start()
        try:
            for peer_rank, peer_path in enumerate(paths):
                if peer_rank == rank:
                    continue
                with socket.socket(socket.AF_UNIX, sock_kind) as sock:
                    sock.settimeout(_FD_SEND_TIMEOUT_S)

View on GitHub (pinned to 0132848349)

Solutions

1. Verify distributed env config (RANK, WORLD_SIZE, MASTER_ADDR) is consistent across ranks.
2. Check for rank restart/retry logic that re-runs exchange_posix_fds against a live receiver.
3. Rerun the job from a clean start if ranks were partially restarted.

assert len({(r, b) for r, b, _ in packets}) == len(packets), "duplicate (rank, base_idx) before install"

try:
    received = exchange_posix_fds(...)
except RuntimeError as e:
    if "duplicate fd" in str(e):
        raise RuntimeError("rank restart/protocol violation — rerun job cleanly") from e
    raise

When it happens

Trigger: recv_loop receives two packets with the same (src_rank, base_idx) pair — a peer rank sending its fd list twice, or multiple senders claiming the same rank id.

Common situations: Ranks restarted mid-exchange resending fds, rank-id collision due to misconfigured distributed env (MASTER_ADDR/RANK/WORLD_SIZE mismatch), or duplicate sockets connecting to the exchange server.

Related errors

• timed out waiting for POSIX fd exchange
• POSIX fd exchange receive failed
• received truncated fd header: {len(data)} < {_FD_INDEX_STRUC
• expected one fd, got {len(fds)}
• world_size must be positive and divide global_heads

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/e3ca8f5e6619eb78. Report an issue: GitHub.