{"record":{"id":"4f25fd5cd722b4ae","repo":"sgl-project/sglang","slug":"sendmsg-sent-sent-bytes-expected-len-header","errorCode":null,"errorMessage":"sendmsg sent {sent} bytes, expected {len(header)}","messagePattern":"sendmsg sent (.+?) bytes, expected (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/utils/cuda_vmm_utils.py","lineNumber":681,"sourceCode":"    Pops from ``mappings`` so a partially-released list is safe to retry.\n    \"\"\"\n    drv = _get_cuda_driver()\n    while mappings:\n        va, span_size, mapped_chunks = mappings.pop()\n        for rel, size in mapped_chunks:\n            check_drv(drv.cuMemUnmap(int(va) + int(rel), int(size)), \"cuMemUnmap\")\n        check_drv(drv.cuMemAddressFree(int(va), int(span_size)), \"cuMemAddressFree\")\n\n\ndef _send_fd(sock, fd: int, src_rank: int, base_idx: int) -> None:\n    fds = array.array(\"i\", [int(fd)])\n    header = struct.pack(\"<QQQ\", int(src_rank), int(base_idx), 1)\n    sent = sock.sendmsg(\n        [header],\n        [(socket.SOL_SOCKET, socket.SCM_RIGHTS, fds.tobytes())],\n    )\n    if sent != len(header):\n        raise RuntimeError(f\"sendmsg sent {sent} bytes, expected {len(header)}\")\n\n\ndef _recv_fd(sock):\n    fd_item_size = array.array(\"i\").itemsize\n    data, ancdata, _, _ = sock.recvmsg(\n        _FD_HEADER_BYTES, socket.CMSG_SPACE(fd_item_size)\n    )\n    if not data:\n        return None\n    if len(data) != _FD_HEADER_BYTES:\n        raise RuntimeError(\n            f\"received truncated fd header: {len(data)} < {_FD_HEADER_BYTES}\"\n        )\n    src_rank, base_idx, fd_count = struct.unpack(\"<QQQ\", data)\n    fds = array.array(\"i\")\n    for level, cmsg_type, cmsg_data in ancdata:\n        if level == socket.SOL_SOCKET and cmsg_type == socket.SCM_RIGHTS:\n            fds.frombytes(cmsg_data[: len(cmsg_data) - (len(cmsg_data) % fd_item_size)])","sourceCodeStart":663,"sourceCodeEnd":699,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/cuda_vmm_utils.py#L663-L699","documentation":"_send_fd uses socket.sendmsg to transmit a 24-byte header plus one SCM_RIGHTS ancillary fd; the kernel normally sends all bytes atomically, so a short send indicates the socket is in a non-blocking/closed state or otherwise degraded.","triggerScenarios":"sock.sendmsg returning fewer than len(header) (24) bytes when sending the (src_rank, base_idx, fd_count) header with an attached fd — non-blocking socket with full buffer, or peer closed mid-send.","commonSituations":"A peer rank crashes during POSIX fd exchange, socket timeouts, or the AF_UNIX socket being put in non-blocking mode unexpectedly during multi-rank startup.","solutions":["Check whether the peer process crashed (look at other ranks' logs) — restart the job.","Retry the send on short sends when the socket is non-blocking, or use a blocking socket with SO_SNDTIMEO.","Verify the AF_UNIX socket path is valid and both endpoints are alive before exchange."],"exampleFix":"// before\nsent = sock.sendmsg([header], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, fds.tobytes())])\n\n// after\nview = memoryview(header)\nwhile view:\n    sent = sock.sendmsg([view], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, fds.tobytes())])\n    if sent == 0:\n        raise RuntimeError(\"peer closed connection\")\n    view = view[sent:]","handlingStrategy":"retry","validationCode":"sock.settimeout(_FD_SEND_TIMEOUT_S)  # blocking with timeout before sendmsg","typeGuard":null,"tryCatchPattern":"for attempt in range(3):\n    try:\n        _send_fd(sock, fd, rank, base_idx)\n        break\n    except RuntimeError as e:\n        if \"sendmsg sent\" not in str(e) or attempt == 2:\n            raise","preventionTips":["Use blocking sockets with a send timeout for fd exchange","Check peer liveness before starting the exchange"],"tags":["network","unix-socket","scm-rights","fd-passing"],"backgroundTag":"socket-send-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}