sgl-project/sglang · error · RuntimeError

received truncated fd header: {len(data)} < {_FD_HEADER_BYTE

Error message

received truncated fd header: {len(data)} < {_FD_HEADER_BYTES}

What it means

_recv_fd expects exactly _FD_HEADER_BYTES (24) bytes of header from recvmsg; receiving a non-empty but shorter datagram means the peer sent a malformed/truncated header, indicating protocol mismatch or corruption on the AF_UNIX channel.

Source

Thrown at python/sglang/srt/utils/cuda_vmm_utils.py:692

    fds = array.array("i", [int(fd)])
    header = struct.pack("<QQQ", int(src_rank), int(base_idx), 1)
    sent = sock.sendmsg(
        [header],
        [(socket.SOL_SOCKET, socket.SCM_RIGHTS, fds.tobytes())],
    )
    if sent != len(header):
        raise RuntimeError(f"sendmsg sent {sent} bytes, expected {len(header)}")


def _recv_fd(sock):
    fd_item_size = array.array("i").itemsize
    data, ancdata, _, _ = sock.recvmsg(
        _FD_HEADER_BYTES, socket.CMSG_SPACE(fd_item_size)
    )
    if not data:
        return None
    if len(data) != _FD_HEADER_BYTES:
        raise RuntimeError(
            f"received truncated fd header: {len(data)} < {_FD_HEADER_BYTES}"
        )
    src_rank, base_idx, fd_count = struct.unpack("<QQQ", data)
    fds = array.array("i")
    for level, cmsg_type, cmsg_data in ancdata:
        if level == socket.SOL_SOCKET and cmsg_type == socket.SCM_RIGHTS:
            fds.frombytes(cmsg_data[: len(cmsg_data) - (len(cmsg_data) % fd_item_size)])
    if fd_count != 1 or len(fds) != 1:
        for fd in fds:
            os.close(fd)
        raise RuntimeError(
            f"expected one fd, got header={fd_count}, ancillary={len(fds)}"
        )
    return int(src_rank), int(base_idx), int(fds[0])


def export_shareable_handles(retained_handles, group: ProcessGroup, rank: int):
    """Export retained VMM handles, preferring FABRIC and falling back to POSIX fds.

View on GitHub (pinned to 0132848349)

Solutions

1. Verify all ranks run the same sglang version (header layout must match).
2. Ensure the socket path is private per job (unique tmpdir) so foreign processes can't connect.
3. Restart the job if corruption is suspected from a partial crash.

try:
    packet = _recv_fd(sock)
except RuntimeError as e:
    if "truncated fd header" in str(e):
        raise RuntimeError("fd exchange protocol mismatch — check rank versions") from e
    raise

When it happens

Trigger: recvmsg returns 0 < len(data) < _FD_HEADER_BYTES on the fd-exchange socket — e.g. peer running a different protocol version, or partial datagram corruption.

Common situations: Mixed-version cluster where ranks run different sglang builds with different header layouts, or stray/foreign processes connecting to the exchange socket path.

Related errors

• sendmsg sent {sent} bytes, expected {len(header)}
• expected one fd, got header={fd_count}, ancillary={len(fds)}
• timed out waiting for POSIX fd exchange
• Failed to get server info. {error_data['error']['message']}
• Failed to get model info: {str(e)}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b9fba2c378e97081. Report an issue: GitHub.