sgl-project/sglang · error · RuntimeError

sendmsg sent {sent} bytes, expected {len(header)}

Error message

sendmsg sent {sent} bytes, expected {len(header)}

What it means

_send_fd uses socket.sendmsg to transmit a 24-byte header plus one SCM_RIGHTS ancillary fd; the kernel normally sends all bytes atomically, so a short send indicates the socket is in a non-blocking/closed state or otherwise degraded.

Source

Thrown at python/sglang/srt/utils/cuda_vmm_utils.py:681

    Pops from ``mappings`` so a partially-released list is safe to retry.
    """
    drv = _get_cuda_driver()
    while mappings:
        va, span_size, mapped_chunks = mappings.pop()
        for rel, size in mapped_chunks:
            check_drv(drv.cuMemUnmap(int(va) + int(rel), int(size)), "cuMemUnmap")
        check_drv(drv.cuMemAddressFree(int(va), int(span_size)), "cuMemAddressFree")


def _send_fd(sock, fd: int, src_rank: int, base_idx: int) -> None:
    fds = array.array("i", [int(fd)])
    header = struct.pack("<QQQ", int(src_rank), int(base_idx), 1)
    sent = sock.sendmsg(
        [header],
        [(socket.SOL_SOCKET, socket.SCM_RIGHTS, fds.tobytes())],
    )
    if sent != len(header):
        raise RuntimeError(f"sendmsg sent {sent} bytes, expected {len(header)}")


def _recv_fd(sock):
    fd_item_size = array.array("i").itemsize
    data, ancdata, _, _ = sock.recvmsg(
        _FD_HEADER_BYTES, socket.CMSG_SPACE(fd_item_size)
    )
    if not data:
        return None
    if len(data) != _FD_HEADER_BYTES:
        raise RuntimeError(
            f"received truncated fd header: {len(data)} < {_FD_HEADER_BYTES}"
        )
    src_rank, base_idx, fd_count = struct.unpack("<QQQ", data)
    fds = array.array("i")
    for level, cmsg_type, cmsg_data in ancdata:
        if level == socket.SOL_SOCKET and cmsg_type == socket.SCM_RIGHTS:
            fds.frombytes(cmsg_data[: len(cmsg_data) - (len(cmsg_data) % fd_item_size)])

View on GitHub (pinned to 0132848349)

Solutions

1. Check whether the peer process crashed (look at other ranks' logs) — restart the job.
2. Retry the send on short sends when the socket is non-blocking, or use a blocking socket with SO_SNDTIMEO.
3. Verify the AF_UNIX socket path is valid and both endpoints are alive before exchange.

Example fix

// before
sent = sock.sendmsg([header], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, fds.tobytes())])

// after
view = memoryview(header)
while view:
    sent = sock.sendmsg([view], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, fds.tobytes())])
    if sent == 0:
        raise RuntimeError("peer closed connection")
    view = view[sent:]

sock.settimeout(_FD_SEND_TIMEOUT_S)  # blocking with timeout before sendmsg

for attempt in range(3):
    try:
        _send_fd(sock, fd, rank, base_idx)
        break
    except RuntimeError as e:
        if "sendmsg sent" not in str(e) or attempt == 2:
            raise

When it happens

Trigger: sock.sendmsg returning fewer than len(header) (24) bytes when sending the (src_rank, base_idx, fd_count) header with an attached fd — non-blocking socket with full buffer, or peer closed mid-send.

Common situations: A peer rank crashes during POSIX fd exchange, socket timeouts, or the AF_UNIX socket being put in non-blocking mode unexpectedly during multi-rank startup.

Related errors

• received truncated fd header: {len(data)} < {_FD_HEADER_BYTE
• expected one fd, got header={fd_count}, ancillary={len(fds)}
• timed out waiting for POSIX fd exchange
• Failed to get server info. {error_data['error']['message']}
• Failed to get model info: {str(e)}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/4f25fd5cd722b4ae. Report an issue: GitHub.