sgl-project/sglang · error · RuntimeError

sendmsg sent {sent} bytes, expected {len(payload)}

Error message

sendmsg sent {sent} bytes, expected {len(payload)}

What it means

_send_fd used sendmsg to pass a file descriptor and it transmitted fewer bytes than the 4-byte fd-index payload. Should not happen on a healthy socket; indicates a partial write or closed connection.

Source

Thrown at python/sglang/srt/weight_cache/transport.py:35

from .protocol import send_msg

logger = logging.getLogger(__name__)

TORCH_IPC_BACKEND = "torch_ipc"
VMM_FD_BACKEND = "vmm_fd"

_FD_INDEX_STRUCT = struct.Struct("<Q")


def _send_fd(sock: socket.socket, fd: int, index: int) -> None:
    payload = _FD_INDEX_STRUCT.pack(index)
    fds = array.array("i", [int(fd)])
    sent = sock.sendmsg(
        [payload], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, fds.tobytes())]
    )
    if sent != len(payload):
        raise RuntimeError(f"sendmsg sent {sent} bytes, expected {len(payload)}")


def _recv_fd(sock: socket.socket) -> Tuple[int, int]:
    fd_item_size = array.array("i").itemsize
    data, ancdata, _, _ = sock.recvmsg(
        _FD_INDEX_STRUCT.size, socket.CMSG_SPACE(fd_item_size)
    )
    if len(data) != _FD_INDEX_STRUCT.size:
        raise RuntimeError(
            f"received truncated fd header: {len(data)} < {_FD_INDEX_STRUCT.size}"
        )
    index = _FD_INDEX_STRUCT.unpack(data)[0]
    fds = array.array("i")
    for level, cmsg_type, cmsg_data in ancdata:
        if level == socket.SOL_SOCKET and cmsg_type == socket.SCM_RIGHTS:
            fds.frombytes(cmsg_data[: len(cmsg_data) - (len(cmsg_data) % fd_item_size)])
    if len(fds) != 1:
        for fd in fds:

View on GitHub (pinned to 0132848349)

Solutions

1. Verify peer liveness before transferring FDs
2. Retry the whole transfer after restarting the daemon
3. Check for protocol misuse (sending on a closed socket)

try:
    _send_fd(sock, index, fd)
except RuntimeError:
    restart_daemon_and_retry()

When it happens

Trigger: Sending an FD over the weight cache transport when the peer socket is closed or the kernel did a short send.

Common situations: Daemon exited while the client was sending tensors; abrupt socket shutdown during FD transfer.

Related errors

• [IpcModelLoader] Error communicating with daemon at {self.so
• [weight_cache:{where}] quantization method {quant_method!r}
• received truncated fd header: {len(data)} < {_FD_INDEX_STRUC
• expected one fd, got {len(fds)}
• Multi-node weight cache daemons (nnodes > 1) require --dist-

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/602e6f757efe1dec. Report an issue: GitHub.