sgl-project/sglang · error · RuntimeError

cuMemGetAllocationGranularity failed for FlashInfer workspac

Error message

cuMemGetAllocationGranularity failed for FlashInfer workspace preflight: {err}

What it means

During a workspace memory preflight for FlashInfer trtllm allreduce fusion, SGLang queries the CUDA driver's cuMemGetAllocationGranularity to compute aligned allocation sizes. If the driver API returns a non-CUDA_SUCCESS error, this RuntimeError propagates the raw CUresult code. It usually indicates a CUDA driver call failure (invalid allocation properties, driver/device mismatch, or corrupted CUDA state).

Source

Thrown at python/sglang/srt/layers/flashinfer_comm_fusion.py:275

    )
    lamport_buffer_size = lamport_comm_size * 3

    # trtllm_create_ipc_workspace_for_all_reduce_fusion rounds each logical
    # buffer to 2 MiB before passing it to SymmDeviceMemory.
    buffer_sizes = (
        ceil_align(size, 1 << 21)
        for size in (buffer_size, flag_size, lamport_buffer_size)
    )

    signal_pad_size = 2048
    allocation_sizes = []
    for buffer_size in buffer_sizes:
        err, alloc_granularity = cuda_driver.cuMemGetAllocationGranularity(
            prop,
            cuda_driver.CUmemAllocationGranularity_flags.CU_MEM_ALLOC_GRANULARITY_RECOMMENDED,
        )
        if err != cuda_driver.CUresult.CUDA_SUCCESS:
            raise RuntimeError(
                "cuMemGetAllocationGranularity failed for FlashInfer "
                f"workspace preflight: {err}"
            )

        allocation_size = ceil_align(buffer_size + signal_pad_size, alloc_granularity)

        mc_prop = cuda_driver.CUmulticastObjectProp()
        mc_prop.numDevices = world_size
        mc_prop.size = allocation_size
        mc_prop.handleTypes = prop.requestedHandleTypes

        err, mc_granularity = cuda_driver.cuMulticastGetGranularity(
            mc_prop,
            cuda_driver.CUmulticastGranularity_flags.CU_MULTICAST_GRANULARITY_RECOMMENDED,
        )
        if err != cuda_driver.CUresult.CUDA_SUCCESS:
            raise RuntimeError(
                "cuMulticastGetGranularity failed for FlashInfer "

View on GitHub (pinned to 0132848349)

Solutions

  1. Upgrade the NVIDIA driver to a version supporting CUDA VMM allocation queries for your GPU (Blackwell/Hopper features need recent drivers)
  2. Check the {err} CUresult code in the message to identify the specific driver failure
  3. Set an environment override / config to skip or disable the workspace preflight if available, or disable flashinfer allreduce fusion
  4. Verify dmesg / nvidia-smi for GPU or driver health issues before retrying
Defensive patterns

Strategy: try-catch

Validate before calling

import ctypes
cudart = ctypes.CDLL("libcudart.so")
ver = ctypes.c_int()
assert cudart.cudaDriverGetVersion(ctypes.byref(ver)) == 0 and ver.value >= 12040, "driver too old for VMM granularity queries"

Try / catch

try:
    sizes = _flashinfer_trtllm_workspace_allocation_sizes(...)
except RuntimeError as e:
    if "cuMemGetAllocationGranularity" in str(e):
        logger.error("driver rejected granularity query; disabling allreduce fusion preflight")
        disable_flashinfer_allreduce_fusion()
    else:
        raise

Prevention

When it happens

Trigger: Calling _preflight_check_workspace_memory -> _flashinfer_trtllm_workspace_allocation_sizes when enabling trtllm allreduce fusion; the driver rejects the CUmemAllocationProp passed to cuMemGetAllocationGranularity (e.g. invalid location/type, unsupported allocator features on old drivers).

Common situations: Old NVIDIA driver versions that predate the memory-allocation-granularity query for the requested prop type; mixed driver/CUDA toolkit versions; passing a prop configured for multicast/VMM features unsupported by the device.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a3d271cc4d28bb2b. Report an issue: GitHub.