{"record":{"id":"37620a2146e0da8c","repo":"sgl-project/sglang","slug":"mxfp8-dense-gemm-requested-via-fp8-gemm-backend","errorCode":null,"errorMessage":"MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer_trtllm, but that kernel requires SM100/SM103 GPUs and FlashInfer.","messagePattern":"MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer_trtllm, but that kernel requires SM100/SM103 GPUs and FlashInfer\\.","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/fp8_utils.py","lineNumber":578,"sourceCode":"    \"\"\"\n    backend = get_fp8_gemm_runner_backend()\n\n    # Handle explicit backend selection via --fp8-gemm-backend\n    if not backend.is_auto():\n        return _dispatch_explicit_backend(backend)\n\n    # Auto mode: Select based purely on hardware/backend availability\n    return _dispatch_auto_backend()\n\n\ndef resolve_mxfp8_dense_gemm_backend() -> Mxfp8DenseGemmBackend:\n    \"\"\"Pick the MXFP8 dense linear backend, honoring `--fp8-gemm-backend` only when it\n    names a backend that owns an MXFP8 dense kernel.\"\"\"\n    backend = get_fp8_gemm_runner_backend()\n\n    if backend.is_flashinfer_trtllm():\n        if not (_is_sm100_supported and is_flashinfer_available()):\n            raise RuntimeError(\n                \"MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer_trtllm, \"\n                \"but that kernel requires SM100/SM103 GPUs and FlashInfer.\"\n            )\n        return Mxfp8DenseGemmBackend.FLASHINFER_TRTLLM\n\n    if backend.is_flashinfer_cutedsl():\n        if not (\n            is_blackwell_supported()\n            and is_flashinfer_available()\n            and _raw_flashinfer_mm_mxfp8.is_backend_supported(\n                \"cute-dsl\", get_device_sm()\n            )\n        ):\n            raise RuntimeError(\n                \"MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer_cutedsl, \"\n                \"but that kernel requires an SM100/SM103 GPU and FlashInfer.\"\n            )\n        return Mxfp8DenseGemmBackend.FLASHINFER_CUTEDSL","sourceCodeStart":560,"sourceCodeEnd":596,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/fp8_utils.py#L560-L596","documentation":"resolve_mxfp8_dense_gemm_backend honors an explicit --fp8-gemm-backend=flashinfer_trtllm only if the hardware is SM100/SM103 (Blackwell) and FlashInfer is installed; otherwise the TRT-LLM MXFP8 dense GEMM cannot run and it raises rather than falling back (explicit user request must be honored or fail loudly).","triggerScenarios":"Setting --fp8-gemm-backend=flashinfer_trtllm on an H100/A100 (not SM100) or in an environment without flashinfer installed, while loading an MXFP8 model — the backend resolution runs at layer init/dispatch time.","commonSituations":"Copying launch flags from Blackwell setups to Hopper clusters; container images that omit flashinfer; CI runs on older GPUs with pinned flags.","solutions":["Drop --fp8-gemm-backend (use auto) and let SGLang pick a backend valid for the GPU","Install FlashInfer and run on an SM100/SM103 (Blackwell) GPU","Switch to a backend available on your hardware, e.g. flashinfer_cutlass or deep_gemm per resolution logic"],"exampleFix":"# before\npython -m sglang.launch_server --model <mxfp8> --fp8-gemm-backend flashinfer_trtllm  # on H100\n# after\npython -m sglang.launch_server --model <mxfp8>  # auto-select","handlingStrategy":"validation","validationCode":"import torch\nfrom sglang.srt.utils import is_flashinfer_available\ncap = torch.cuda.get_device_capability(0)[0] if torch.cuda.is_available() else 0\nif args.fp8_gemm_backend == \"flashinfer_trtllm\":\n    assert cap >= 10 and is_flashinfer_available(), \"flashinfer_trtllm mxfp8 needs SM100+ and FlashInfer; use auto\"","typeGuard":"def trtllm_mxfp8_available() -> bool:\n    import torch\n    from sglang.srt.utils import is_flashinfer_available\n    return torch.cuda.is_available() and torch.cuda.get_device_capability(0)[0] >= 10 and is_flashinfer_available()","tryCatchPattern":null,"preventionTips":["Don't copy Blackwell-specific flags onto Hopper clusters","Install flashinfer in serving images","Default to auto backend selection"],"tags":["quantization","mxfp8","gemm-backend","hardware-compatibility","flashinfer"],"backgroundTag":"gemm-backend-hardware-unsupported","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}