{"record":{"id":"4f43a78bf21a0d58","repo":"sgl-project/sglang","slug":"vmm-handle-export-failed-fabric-export-failed-on","errorCode":null,"errorMessage":"VMM handle export failed: FABRIC export failed on at least one rank and POSIX fd export failed on at least one rank","messagePattern":"VMM handle export failed: FABRIC export failed on at least one rank and POSIX fd export failed on at least one rank","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/utils/cuda_vmm_utils.py","lineNumber":772,"sourceCode":"    except Exception as e:\n        posix_error = e\n        posix_ok = False\n        for fd in posix_fds:\n            try:\n                os.close(fd)\n            except OSError:\n                pass\n        posix_fds = []\n\n    if not all_ranks_ok(group, posix_ok):\n        cause = posix_error or fabric_error\n        message = (\n            \"VMM handle export failed: FABRIC export failed on at least one \"\n            \"rank and POSIX fd export failed on at least one rank\"\n        )\n        if cause is not None:\n            message += f\"; local rank {rank} error: {cause}\"\n        raise RuntimeError(message) from posix_error\n\n    return [], posix_fds, False\n\n\ndef exchange_posix_fds(\n    group: ProcessGroup,\n    rank: int,\n    world_size: int,\n    local_fds: List[int],\n    peer_base_counts: List[int],\n):\n    \"\"\"Exchange POSIX file descriptors across ranks via SCM_RIGHTS over a UNIX\n    socket. Returns ``{(src_rank, base_idx): fd}`` for every peer. The caller\n    owns the received fds and must close them.\n    \"\"\"\n    sock_kind = socket.SOCK_SEQPACKET\n    sock_dir = tempfile.mkdtemp(prefix=\"sgl_ar_fd_\")\n    sock_path = os.path.join(sock_dir, f\"rank_{rank}.sock\")","sourceCodeStart":754,"sourceCodeEnd":790,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/cuda_vmm_utils.py#L754-L790","documentation":"export_shareable_handles tries two export mechanisms for CUDA VMM handles: NVSwitch FABRIC export and POSIX fd export. If at least one rank fails FABRIC export AND at least one rank fails POSIX fd export, there is no common fallback, so it raises with the local rank's cause chained.","triggerScenarios":"Calling export_shareable_handles in a multi-rank job where FABRIC (cuMemExportToShareableHandle with CU_MEM_HANDLE_TYPE_FABRIC) fails on some rank and POSIX fd export fails on another — e.g. GPUs without fabric support mixed with a container lacking /dev/fabric or fd permissions.","commonSituations":"Heterogeneous nodes (some with NVSwitch, some PCIe-only), older drivers lacking fabric handle support, containers missing device nodes, or IMEX channels not configured.","solutions":["Check the chained `cause` in the traceback to see which export path failed locally and why (driver version, permission, unsupported handle type).","Prefer running P2P-sharing ranks on homogeneous NVSwitch nodes with a driver that supports fabric handles.","Ensure the container exposes /dev/* fabric device nodes and has CAP_SYS_ADMIN or appropriate permissions for fd export."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"# probe capabilities before the multi-rank export\nimport ctypes\nfrom python.sglang.srt.utils.cuda_vmm_utils import _get_cuda_driver\n# ensure driver >= fabric-handle support and /dev devices exist before launching job","typeGuard":null,"tryCatchPattern":"try:\n    fabric_fds, posix_fds, use_fabric = export_shareable_handles(handles, group, rank)\nexcept RuntimeError as e:\n    if \"VMM handle export failed\" in str(e):\n        # fall back to non-shared / per-rank memory strategy\n        disable_peer_mapping = True\n    else:\n        raise","preventionTips":["Run on homogeneous NVSwitch nodes with recent drivers","Expose fabric device nodes in container spec","Preflight-check fabric export support on all ranks at startup"],"tags":["cuda","distributed","fabric","p2p","driver-support"],"backgroundTag":"cuda-fabric-export-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}