{"record":{"id":"a3d271cc4d28bb2b","repo":"sgl-project/sglang","slug":"cumemgetallocationgranularity-failed-for-flashinfe","errorCode":null,"errorMessage":"cuMemGetAllocationGranularity failed for FlashInfer workspace preflight: {err}","messagePattern":"cuMemGetAllocationGranularity failed for FlashInfer workspace preflight: (.+?)","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/flashinfer_comm_fusion.py","lineNumber":275,"sourceCode":"    )\n    lamport_buffer_size = lamport_comm_size * 3\n\n    # trtllm_create_ipc_workspace_for_all_reduce_fusion rounds each logical\n    # buffer to 2 MiB before passing it to SymmDeviceMemory.\n    buffer_sizes = (\n        ceil_align(size, 1 << 21)\n        for size in (buffer_size, flag_size, lamport_buffer_size)\n    )\n\n    signal_pad_size = 2048\n    allocation_sizes = []\n    for buffer_size in buffer_sizes:\n        err, alloc_granularity = cuda_driver.cuMemGetAllocationGranularity(\n            prop,\n            cuda_driver.CUmemAllocationGranularity_flags.CU_MEM_ALLOC_GRANULARITY_RECOMMENDED,\n        )\n        if err != cuda_driver.CUresult.CUDA_SUCCESS:\n            raise RuntimeError(\n                \"cuMemGetAllocationGranularity failed for FlashInfer \"\n                f\"workspace preflight: {err}\"\n            )\n\n        allocation_size = ceil_align(buffer_size + signal_pad_size, alloc_granularity)\n\n        mc_prop = cuda_driver.CUmulticastObjectProp()\n        mc_prop.numDevices = world_size\n        mc_prop.size = allocation_size\n        mc_prop.handleTypes = prop.requestedHandleTypes\n\n        err, mc_granularity = cuda_driver.cuMulticastGetGranularity(\n            mc_prop,\n            cuda_driver.CUmulticastGranularity_flags.CU_MULTICAST_GRANULARITY_RECOMMENDED,\n        )\n        if err != cuda_driver.CUresult.CUDA_SUCCESS:\n            raise RuntimeError(\n                \"cuMulticastGetGranularity failed for FlashInfer \"","sourceCodeStart":257,"sourceCodeEnd":293,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/flashinfer_comm_fusion.py#L257-L293","documentation":"During a workspace memory preflight for FlashInfer trtllm allreduce fusion, SGLang queries the CUDA driver's cuMemGetAllocationGranularity to compute aligned allocation sizes. If the driver API returns a non-CUDA_SUCCESS error, this RuntimeError propagates the raw CUresult code. It usually indicates a CUDA driver call failure (invalid allocation properties, driver/device mismatch, or corrupted CUDA state).","triggerScenarios":"Calling _preflight_check_workspace_memory -> _flashinfer_trtllm_workspace_allocation_sizes when enabling trtllm allreduce fusion; the driver rejects the CUmemAllocationProp passed to cuMemGetAllocationGranularity (e.g. invalid location/type, unsupported allocator features on old drivers).","commonSituations":"Old NVIDIA driver versions that predate the memory-allocation-granularity query for the requested prop type; mixed driver/CUDA toolkit versions; passing a prop configured for multicast/VMM features unsupported by the device.","solutions":["Upgrade the NVIDIA driver to a version supporting CUDA VMM allocation queries for your GPU (Blackwell/Hopper features need recent drivers)","Check the {err} CUresult code in the message to identify the specific driver failure","Set an environment override / config to skip or disable the workspace preflight if available, or disable flashinfer allreduce fusion","Verify dmesg / nvidia-smi for GPU or driver health issues before retrying"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"import ctypes\ncudart = ctypes.CDLL(\"libcudart.so\")\nver = ctypes.c_int()\nassert cudart.cudaDriverGetVersion(ctypes.byref(ver)) == 0 and ver.value >= 12040, \"driver too old for VMM granularity queries\"","typeGuard":null,"tryCatchPattern":"try:\n    sizes = _flashinfer_trtllm_workspace_allocation_sizes(...)\nexcept RuntimeError as e:\n    if \"cuMemGetAllocationGranularity\" in str(e):\n        logger.error(\"driver rejected granularity query; disabling allreduce fusion preflight\")\n        disable_flashinfer_allreduce_fusion()\n    else:\n        raise","preventionTips":["Pin recent NVIDIA drivers in deployment images","Surface the CUresult code in logs for triage","Provide a config switch to skip the preflight on unsupported stacks"],"tags":["cuda-driver","flashinfer","workspace-preflight","granularity","driver-version"],"backgroundTag":"cuda-driver-api-error","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}