What causes "expected one fd, got header={fd_count}, ancillary={len(fds)}" in sgl-project/sglang?

Confirm every rank uses the identical sglang build/protocol version.

sgl-project/sglang · error · RuntimeError

expected one fd, got header={fd_count}, ancillary={len(fds)}

Error message

expected one fd, got header={fd_count}, ancillary={len(fds)}

What it means

After parsing the header, _recv_fd requires exactly one file descriptor: fd_count==1 from the header and exactly one fd in the SCM_RIGHTS ancillary data. Any mismatch (and any received fds are closed) raises this to prevent leaking or misinterpreting fds.

Source

Thrown at python/sglang/srt/utils/cuda_vmm_utils.py:703

    fd_item_size = array.array("i").itemsize
    data, ancdata, _, _ = sock.recvmsg(
        _FD_HEADER_BYTES, socket.CMSG_SPACE(fd_item_size)
    )
    if not data:
        return None
    if len(data) != _FD_HEADER_BYTES:
        raise RuntimeError(
            f"received truncated fd header: {len(data)} < {_FD_HEADER_BYTES}"
        )
    src_rank, base_idx, fd_count = struct.unpack("<QQQ", data)
    fds = array.array("i")
    for level, cmsg_type, cmsg_data in ancdata:
        if level == socket.SOL_SOCKET and cmsg_type == socket.SCM_RIGHTS:
            fds.frombytes(cmsg_data[: len(cmsg_data) - (len(cmsg_data) % fd_item_size)])
    if fd_count != 1 or len(fds) != 1:
        for fd in fds:
            os.close(fd)
        raise RuntimeError(
            f"expected one fd, got header={fd_count}, ancillary={len(fds)}"
        )
    return int(src_rank), int(base_idx), int(fds[0])


def export_shareable_handles(retained_handles, group: ProcessGroup, rank: int):
    """Export retained VMM handles, preferring FABRIC and falling back to POSIX fds.

    FABRIC is used only if every rank can export it; otherwise all ranks use POSIX
    fds. Returns ``(fabric_handles, posix_fds, use_fabric)`` (one list populated);
    raises if both fail on any rank. Caller owns the returned ``posix_fds``.
    """
    drv = _get_cuda_driver()
    FABRIC = drv.CUmemAllocationHandleType.CU_MEM_HANDLE_TYPE_FABRIC
    POSIX_FD = drv.CUmemAllocationHandleType.CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR

    fabric_handles: List[bytes] = []
    fabric_error: Optional[Exception] = None

View on GitHub (pinned to 0132848349)

Solutions

Confirm every rank uses the identical sglang build/protocol version.
Check for local modifications to _send_fd that batch multiple fds into one sendmsg.
Restart the exchange; if persistent, report as a version-skew bug.

Defensive patterns

Strategy: try-catch

Try / catch

try:
    src_rank, base_idx, fd = _recv_fd(sock)
except RuntimeError as e:
    if "expected one fd" in str(e):
        log_protocol_version(); raise

Prevention

Never modify the send side to batch fds without updating the receiver
Verify version homogeneity across ranks

When it happens

Trigger: Header's fd_count != 1, or the ancillary SCM_RIGHTS cmsg carries zero or multiple fds — sender-side bug or version mismatch in the exchange protocol.

Common situations: A rank built with a different _send_fd/_recv_fd protocol (e.g. batching multiple fds per message), or cmsg truncation due to insufficient CMSG_SPACE buffer on the receive side.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f0063782499bf121. Report an issue: GitHub.