sgl-project/sglang · error · RuntimeError

MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer

Error message

MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer_cutedsl, but that kernel requires an SM100/SM103 GPU and FlashInfer.

What it means

--fp8-gemm-backend=flashinfer_cutedsl for MXFP8 dense GEMM requires an SM100/SM103 GPU, FlashInfer installed, and the cute-dsl backend reporting support for the device's SM version (checked via _raw_flashinfer_mm_mxfp8.is_backend_supported). If any check fails, the explicit request raises instead of silently switching kernels.

Source

Thrown at python/sglang/srt/layers/quantization/fp8_utils.py:592

    backend = get_fp8_gemm_runner_backend()

    if backend.is_flashinfer_trtllm():
        if not (_is_sm100_supported and is_flashinfer_available()):
            raise RuntimeError(
                "MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer_trtllm, "
                "but that kernel requires SM100/SM103 GPUs and FlashInfer."
            )
        return Mxfp8DenseGemmBackend.FLASHINFER_TRTLLM

    if backend.is_flashinfer_cutedsl():
        if not (
            is_blackwell_supported()
            and is_flashinfer_available()
            and _raw_flashinfer_mm_mxfp8.is_backend_supported(
                "cute-dsl", get_device_sm()
            )
        ):
            raise RuntimeError(
                "MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer_cutedsl, "
                "but that kernel requires an SM100/SM103 GPU and FlashInfer."
            )
        return Mxfp8DenseGemmBackend.FLASHINFER_CUTEDSL

    if backend.is_flashinfer_cutlass():
        if not (is_blackwell_supported() and is_flashinfer_available()):
            raise RuntimeError(
                "MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer_cutlass, "
                "but that kernel requires Blackwell GPUs and FlashInfer."
            )
        return Mxfp8DenseGemmBackend.FLASHINFER_CUTLASS

    if backend.is_deep_gemm():
        if not deep_gemm_wrapper.ENABLE_JIT_DEEPGEMM:
            raise RuntimeError(
                "MXFP8 dense GEMM requested via --fp8-gemm-backend=deep_gemm, but "
                "DeepGEMM is not available (package missing or "

View on GitHub (pinned to 0132848349)

Solutions

1. Remove --fp8-gemm-backend (auto) so a supported MXFP8 backend is chosen
2. Upgrade FlashInfer to a version with cute-dsl MXFP8 support and run on SM100/SM103
3. Use flashinfer_cutlass or another supported backend on your GPU

Example fix

# before
--fp8-gemm-backend flashinfer_cutedsl  # non-SM100 GPU
# after
--fp8-gemm-backend auto

import torch
if args.fp8_gemm_backend == "flashinfer_cutedsl":
    from sglang.srt.utils import is_flashinfer_available
    assert torch.cuda.is_available() and torch.cuda.get_device_capability(0)[0] >= 10 and is_flashinfer_available(), \
        "flashinfer_cutedsl mxfp8 needs SM100/SM103 + FlashInfer; use auto"

def cutedsl_mxfp8_available() -> bool:
    import torch
    try:
        from sglang.srt.utils import is_flashinfer_available
        return torch.cuda.is_available() and torch.cuda.get_device_capability(0)[0] >= 10 and is_flashinfer_available()
    except ImportError:
        return False

When it happens

Trigger: Setting --fp8-gemm-backend=flashinfer_cutedsl on a non-Blackwell GPU, without FlashInfer, or on an SM version the cute-dsl kernel doesn't support, while resolving the MXFP8 dense backend during layer init or dispatch.

Common situations: Older flashinfer wheels lacking cute-dsl MXFP8 kernels; running on Hopper with flags copied from Blackwell docs; nightly-dependent cute-dsl support gaps.

Related errors

• MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer
• --fp8-gemm-backend=deep_gemm cannot serve MXFP8 weight shape
• MXFP8 MoE quantization requires SM100 or ROCm gfx95 (gfx942
• MXFP8 fused prologue requires head_dim-aligned Q/K/V.
• MXFP8 fused prologue requires K/V scale buffers.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/e9605912d4f5d59d. Report an issue: GitHub.