sgl-project/sglang · error · RuntimeError

cuMulticastGetGranularity failed for FlashInfer workspace pr

Error message

cuMulticastGetGranularity failed for FlashInfer workspace preflight: {err}

What it means

When preflighting workspace memory for FlashInfer trtllm allreduce fusion, SGLang calls cuMulticastGetGranularity to align allocation sizes to the multicast granularity. If the CUDA driver returns an error other than CUDA_SUCCESS, this RuntimeError is raised with the raw CUresult. Multicast granularity queries require driver support for multicast objects (typically NVSwitch-based Hopper/Blackwell systems), so failure often means the platform or driver lacks multicast support.

Source

Thrown at python/sglang/srt/layers/flashinfer_comm_fusion.py:292

        if err != cuda_driver.CUresult.CUDA_SUCCESS:
            raise RuntimeError(
                "cuMemGetAllocationGranularity failed for FlashInfer "
                f"workspace preflight: {err}"
            )

        allocation_size = ceil_align(buffer_size + signal_pad_size, alloc_granularity)

        mc_prop = cuda_driver.CUmulticastObjectProp()
        mc_prop.numDevices = world_size
        mc_prop.size = allocation_size
        mc_prop.handleTypes = prop.requestedHandleTypes

        err, mc_granularity = cuda_driver.cuMulticastGetGranularity(
            mc_prop,
            cuda_driver.CUmulticastGranularity_flags.CU_MULTICAST_GRANULARITY_RECOMMENDED,
        )
        if err != cuda_driver.CUresult.CUDA_SUCCESS:
            raise RuntimeError(
                "cuMulticastGetGranularity failed for FlashInfer "
                f"workspace preflight: {err}"
            )

        allocation_size = ceil_align(allocation_size, mc_granularity)
        allocation_sizes.append(allocation_size)
    return allocation_sizes


def _probe_cumem_create_sequence(cuda_driver, allocation_sizes, prop) -> bool:
    handles = []
    try:
        for allocation_size in allocation_sizes:
            err, handle = cuda_driver.cuMemCreate(allocation_size, prop, 0)
            if err != cuda_driver.CUresult.CUDA_SUCCESS:
                return False
            handles.append(handle)
        return True

View on GitHub (pinned to 0132848349)

Solutions

Check the {err} CUresult code to confirm multicast-unsupported vs driver error
Disable flashinfer allreduce fusion / trtllm backend on this topology (fall back to standard allreduce)
Upgrade the NVIDIA driver to one supporting multicast objects for your GPU/driver stack
Run on NVSwitch-connected Hopper/Blackwell nodes when multicast fusion is required

Defensive patterns

Strategy: try-catch

Validate before calling

# check multicast support before enabling trtllm fusion
import subprocess
 topo = subprocess.run(["nvidia-smi","topo","-m"], capture_output=True, text=True).stdout
assert "NV" in topo, "no NVLink/NVSwitch fabric; multicast granularity query will fail"

Try / catch

try:
    _preflight_check_workspace_memory(...)
except RuntimeError as e:
    if "cuMulticastGetGranularity" in str(e):
        logger.warning("multicast unsupported on this node; falling back to standard allreduce")
        use_flashinfer_allreduce_fusion = False
    else:
        raise

Prevention

Only enable multicast-based fusion on NVSwitch Hopper/Blackwell nodes
Keep a fallback allreduce path in server config
Smoke-test the preflight in CI on target hardware

When it happens

Trigger: Enabling trtllm allreduce fusion workspace preflight on a system whose driver or topology does not support multicast memory (no NVSwitch, single GPU, or driver too old), making cuMulticastGetGranularity(mc_prop, CU_MULTICAST_GRANULARITY_RECOMMENDED) fail.

Common situations: Running flashinfer allreduce fusion on non-NVSwitch multi-GPU nodes (e.g. PCIe-only topology), dev machines with a single GPU, or drivers predating CUmulticastObject support.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8eb719c31622c3cd. Report an issue: GitHub.