sgl-project/sglang · error · RuntimeError

MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer

Error message

MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer_trtllm, but that kernel requires SM100/SM103 GPUs and FlashInfer.

What it means

resolve_mxfp8_dense_gemm_backend honors an explicit --fp8-gemm-backend=flashinfer_trtllm only if the hardware is SM100/SM103 (Blackwell) and FlashInfer is installed; otherwise the TRT-LLM MXFP8 dense GEMM cannot run and it raises rather than falling back (explicit user request must be honored or fail loudly).

Source

Thrown at python/sglang/srt/layers/quantization/fp8_utils.py:578

    """
    backend = get_fp8_gemm_runner_backend()

    # Handle explicit backend selection via --fp8-gemm-backend
    if not backend.is_auto():
        return _dispatch_explicit_backend(backend)

    # Auto mode: Select based purely on hardware/backend availability
    return _dispatch_auto_backend()


def resolve_mxfp8_dense_gemm_backend() -> Mxfp8DenseGemmBackend:
    """Pick the MXFP8 dense linear backend, honoring `--fp8-gemm-backend` only when it
    names a backend that owns an MXFP8 dense kernel."""
    backend = get_fp8_gemm_runner_backend()

    if backend.is_flashinfer_trtllm():
        if not (_is_sm100_supported and is_flashinfer_available()):
            raise RuntimeError(
                "MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer_trtllm, "
                "but that kernel requires SM100/SM103 GPUs and FlashInfer."
            )
        return Mxfp8DenseGemmBackend.FLASHINFER_TRTLLM

    if backend.is_flashinfer_cutedsl():
        if not (
            is_blackwell_supported()
            and is_flashinfer_available()
            and _raw_flashinfer_mm_mxfp8.is_backend_supported(
                "cute-dsl", get_device_sm()
            )
        ):
            raise RuntimeError(
                "MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer_cutedsl, "
                "but that kernel requires an SM100/SM103 GPU and FlashInfer."
            )
        return Mxfp8DenseGemmBackend.FLASHINFER_CUTEDSL

View on GitHub (pinned to 0132848349)

Solutions

1. Drop --fp8-gemm-backend (use auto) and let SGLang pick a backend valid for the GPU
2. Install FlashInfer and run on an SM100/SM103 (Blackwell) GPU
3. Switch to a backend available on your hardware, e.g. flashinfer_cutlass or deep_gemm per resolution logic

Example fix

# before
python -m sglang.launch_server --model <mxfp8> --fp8-gemm-backend flashinfer_trtllm  # on H100
# after
python -m sglang.launch_server --model <mxfp8>  # auto-select

import torch
from sglang.srt.utils import is_flashinfer_available
cap = torch.cuda.get_device_capability(0)[0] if torch.cuda.is_available() else 0
if args.fp8_gemm_backend == "flashinfer_trtllm":
    assert cap >= 10 and is_flashinfer_available(), "flashinfer_trtllm mxfp8 needs SM100+ and FlashInfer; use auto"

def trtllm_mxfp8_available() -> bool:
    import torch
    from sglang.srt.utils import is_flashinfer_available
    return torch.cuda.is_available() and torch.cuda.get_device_capability(0)[0] >= 10 and is_flashinfer_available()

When it happens

Trigger: Setting --fp8-gemm-backend=flashinfer_trtllm on an H100/A100 (not SM100) or in an environment without flashinfer installed, while loading an MXFP8 model — the backend resolution runs at layer init/dispatch time.

Common situations: Copying launch flags from Blackwell setups to Hopper clusters; container images that omit flashinfer; CI runs on older GPUs with pinned flags.

Related errors

• MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer
• --fp8-gemm-backend=deep_gemm cannot serve MXFP8 weight shape
• MXFP8 MoE quantization requires SM100 or ROCm gfx95 (gfx942
• MXFP8 fused prologue requires head_dim-aligned Q/K/V.
• MXFP8 fused prologue requires K/V scale buffers.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/37620a2146e0da8c. Report an issue: GitHub.