{"record":{"id":"f0063782499bf121","repo":"sgl-project/sglang","slug":"expected-one-fd-got-header-fd-count-ancillary","errorCode":null,"errorMessage":"expected one fd, got header={fd_count}, ancillary={len(fds)}","messagePattern":"expected one fd, got header=(.+?), ancillary=(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/utils/cuda_vmm_utils.py","lineNumber":703,"sourceCode":"    fd_item_size = array.array(\"i\").itemsize\n    data, ancdata, _, _ = sock.recvmsg(\n        _FD_HEADER_BYTES, socket.CMSG_SPACE(fd_item_size)\n    )\n    if not data:\n        return None\n    if len(data) != _FD_HEADER_BYTES:\n        raise RuntimeError(\n            f\"received truncated fd header: {len(data)} < {_FD_HEADER_BYTES}\"\n        )\n    src_rank, base_idx, fd_count = struct.unpack(\"<QQQ\", data)\n    fds = array.array(\"i\")\n    for level, cmsg_type, cmsg_data in ancdata:\n        if level == socket.SOL_SOCKET and cmsg_type == socket.SCM_RIGHTS:\n            fds.frombytes(cmsg_data[: len(cmsg_data) - (len(cmsg_data) % fd_item_size)])\n    if fd_count != 1 or len(fds) != 1:\n        for fd in fds:\n            os.close(fd)\n        raise RuntimeError(\n            f\"expected one fd, got header={fd_count}, ancillary={len(fds)}\"\n        )\n    return int(src_rank), int(base_idx), int(fds[0])\n\n\ndef export_shareable_handles(retained_handles, group: ProcessGroup, rank: int):\n    \"\"\"Export retained VMM handles, preferring FABRIC and falling back to POSIX fds.\n\n    FABRIC is used only if every rank can export it; otherwise all ranks use POSIX\n    fds. Returns ``(fabric_handles, posix_fds, use_fabric)`` (one list populated);\n    raises if both fail on any rank. Caller owns the returned ``posix_fds``.\n    \"\"\"\n    drv = _get_cuda_driver()\n    FABRIC = drv.CUmemAllocationHandleType.CU_MEM_HANDLE_TYPE_FABRIC\n    POSIX_FD = drv.CUmemAllocationHandleType.CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR\n\n    fabric_handles: List[bytes] = []\n    fabric_error: Optional[Exception] = None","sourceCodeStart":685,"sourceCodeEnd":721,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/cuda_vmm_utils.py#L685-L721","documentation":"After parsing the header, _recv_fd requires exactly one file descriptor: fd_count==1 from the header and exactly one fd in the SCM_RIGHTS ancillary data. Any mismatch (and any received fds are closed) raises this to prevent leaking or misinterpreting fds.","triggerScenarios":"Header's fd_count != 1, or the ancillary SCM_RIGHTS cmsg carries zero or multiple fds — sender-side bug or version mismatch in the exchange protocol.","commonSituations":"A rank built with a different _send_fd/_recv_fd protocol (e.g. batching multiple fds per message), or cmsg truncation due to insufficient CMSG_SPACE buffer on the receive side.","solutions":["Confirm every rank uses the identical sglang build/protocol version.","Check for local modifications to _send_fd that batch multiple fds into one sendmsg.","Restart the exchange; if persistent, report as a version-skew bug."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    src_rank, base_idx, fd = _recv_fd(sock)\nexcept RuntimeError as e:\n    if \"expected one fd\" in str(e):\n        log_protocol_version(); raise","preventionTips":["Never modify the send side to batch fds without updating the receiver","Verify version homogeneity across ranks"],"tags":["network","scm-rights","fd-passing","validation"],"backgroundTag":"protocol-deserialization-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}