{"record":{"id":"8eb719c31622c3cd","repo":"sgl-project/sglang","slug":"cumulticastgetgranularity-failed-for-flashinfer-wo","errorCode":null,"errorMessage":"cuMulticastGetGranularity failed for FlashInfer workspace preflight: {err}","messagePattern":"cuMulticastGetGranularity failed for FlashInfer workspace preflight: (.+?)","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/flashinfer_comm_fusion.py","lineNumber":292,"sourceCode":"        if err != cuda_driver.CUresult.CUDA_SUCCESS:\n            raise RuntimeError(\n                \"cuMemGetAllocationGranularity failed for FlashInfer \"\n                f\"workspace preflight: {err}\"\n            )\n\n        allocation_size = ceil_align(buffer_size + signal_pad_size, alloc_granularity)\n\n        mc_prop = cuda_driver.CUmulticastObjectProp()\n        mc_prop.numDevices = world_size\n        mc_prop.size = allocation_size\n        mc_prop.handleTypes = prop.requestedHandleTypes\n\n        err, mc_granularity = cuda_driver.cuMulticastGetGranularity(\n            mc_prop,\n            cuda_driver.CUmulticastGranularity_flags.CU_MULTICAST_GRANULARITY_RECOMMENDED,\n        )\n        if err != cuda_driver.CUresult.CUDA_SUCCESS:\n            raise RuntimeError(\n                \"cuMulticastGetGranularity failed for FlashInfer \"\n                f\"workspace preflight: {err}\"\n            )\n\n        allocation_size = ceil_align(allocation_size, mc_granularity)\n        allocation_sizes.append(allocation_size)\n    return allocation_sizes\n\n\ndef _probe_cumem_create_sequence(cuda_driver, allocation_sizes, prop) -> bool:\n    handles = []\n    try:\n        for allocation_size in allocation_sizes:\n            err, handle = cuda_driver.cuMemCreate(allocation_size, prop, 0)\n            if err != cuda_driver.CUresult.CUDA_SUCCESS:\n                return False\n            handles.append(handle)\n        return True","sourceCodeStart":274,"sourceCodeEnd":310,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/flashinfer_comm_fusion.py#L274-L310","documentation":"When preflighting workspace memory for FlashInfer trtllm allreduce fusion, SGLang calls cuMulticastGetGranularity to align allocation sizes to the multicast granularity. If the CUDA driver returns an error other than CUDA_SUCCESS, this RuntimeError is raised with the raw CUresult. Multicast granularity queries require driver support for multicast objects (typically NVSwitch-based Hopper/Blackwell systems), so failure often means the platform or driver lacks multicast support.","triggerScenarios":"Enabling trtllm allreduce fusion workspace preflight on a system whose driver or topology does not support multicast memory (no NVSwitch, single GPU, or driver too old), making cuMulticastGetGranularity(mc_prop, CU_MULTICAST_GRANULARITY_RECOMMENDED) fail.","commonSituations":"Running flashinfer allreduce fusion on non-NVSwitch multi-GPU nodes (e.g. PCIe-only topology), dev machines with a single GPU, or drivers predating CUmulticastObject support.","solutions":["Check the {err} CUresult code to confirm multicast-unsupported vs driver error","Disable flashinfer allreduce fusion / trtllm backend on this topology (fall back to standard allreduce)","Upgrade the NVIDIA driver to one supporting multicast objects for your GPU/driver stack","Run on NVSwitch-connected Hopper/Blackwell nodes when multicast fusion is required"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"# check multicast support before enabling trtllm fusion\nimport subprocess\n topo = subprocess.run([\"nvidia-smi\",\"topo\",\"-m\"], capture_output=True, text=True).stdout\nassert \"NV\" in topo, \"no NVLink/NVSwitch fabric; multicast granularity query will fail\"","typeGuard":null,"tryCatchPattern":"try:\n    _preflight_check_workspace_memory(...)\nexcept RuntimeError as e:\n    if \"cuMulticastGetGranularity\" in str(e):\n        logger.warning(\"multicast unsupported on this node; falling back to standard allreduce\")\n        use_flashinfer_allreduce_fusion = False\n    else:\n        raise","preventionTips":["Only enable multicast-based fusion on NVSwitch Hopper/Blackwell nodes","Keep a fallback allreduce path in server config","Smoke-test the preflight in CI on target hardware"],"tags":["cuda-driver","multicast","flashinfer","workspace-preflight","nvswitch"],"backgroundTag":"cuda-driver-api-error","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}