{"record":{"id":"b9fba2c378e97081","repo":"sgl-project/sglang","slug":"received-truncated-fd-header-len-data-fd-h","errorCode":null,"errorMessage":"received truncated fd header: {len(data)} < {_FD_HEADER_BYTES}","messagePattern":"received truncated fd header: (.+?) < (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/utils/cuda_vmm_utils.py","lineNumber":692,"sourceCode":"    fds = array.array(\"i\", [int(fd)])\n    header = struct.pack(\"<QQQ\", int(src_rank), int(base_idx), 1)\n    sent = sock.sendmsg(\n        [header],\n        [(socket.SOL_SOCKET, socket.SCM_RIGHTS, fds.tobytes())],\n    )\n    if sent != len(header):\n        raise RuntimeError(f\"sendmsg sent {sent} bytes, expected {len(header)}\")\n\n\ndef _recv_fd(sock):\n    fd_item_size = array.array(\"i\").itemsize\n    data, ancdata, _, _ = sock.recvmsg(\n        _FD_HEADER_BYTES, socket.CMSG_SPACE(fd_item_size)\n    )\n    if not data:\n        return None\n    if len(data) != _FD_HEADER_BYTES:\n        raise RuntimeError(\n            f\"received truncated fd header: {len(data)} < {_FD_HEADER_BYTES}\"\n        )\n    src_rank, base_idx, fd_count = struct.unpack(\"<QQQ\", data)\n    fds = array.array(\"i\")\n    for level, cmsg_type, cmsg_data in ancdata:\n        if level == socket.SOL_SOCKET and cmsg_type == socket.SCM_RIGHTS:\n            fds.frombytes(cmsg_data[: len(cmsg_data) - (len(cmsg_data) % fd_item_size)])\n    if fd_count != 1 or len(fds) != 1:\n        for fd in fds:\n            os.close(fd)\n        raise RuntimeError(\n            f\"expected one fd, got header={fd_count}, ancillary={len(fds)}\"\n        )\n    return int(src_rank), int(base_idx), int(fds[0])\n\n\ndef export_shareable_handles(retained_handles, group: ProcessGroup, rank: int):\n    \"\"\"Export retained VMM handles, preferring FABRIC and falling back to POSIX fds.","sourceCodeStart":674,"sourceCodeEnd":710,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/cuda_vmm_utils.py#L674-L710","documentation":"_recv_fd expects exactly _FD_HEADER_BYTES (24) bytes of header from recvmsg; receiving a non-empty but shorter datagram means the peer sent a malformed/truncated header, indicating protocol mismatch or corruption on the AF_UNIX channel.","triggerScenarios":"recvmsg returns 0 < len(data) < _FD_HEADER_BYTES on the fd-exchange socket — e.g. peer running a different protocol version, or partial datagram corruption.","commonSituations":"Mixed-version cluster where ranks run different sglang builds with different header layouts, or stray/foreign processes connecting to the exchange socket path.","solutions":["Verify all ranks run the same sglang version (header layout must match).","Ensure the socket path is private per job (unique tmpdir) so foreign processes can't connect.","Restart the job if corruption is suspected from a partial crash."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    packet = _recv_fd(sock)\nexcept RuntimeError as e:\n    if \"truncated fd header\" in str(e):\n        raise RuntimeError(\"fd exchange protocol mismatch — check rank versions\") from e\n    raise","preventionTips":["Pin identical sglang versions on all ranks","Use a job-unique socket path"],"tags":["network","unix-socket","protocol-mismatch","fd-passing"],"backgroundTag":"protocol-deserialization-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}