{"record":{"id":"faf1e961d8246da2","repo":"sgl-project/sglang","slug":"fp8-gemm-backend-deep-gemm-cannot-serve-mxfp8-we","errorCode":null,"errorMessage":"--fp8-gemm-backend=deep_gemm cannot serve MXFP8 weight shape ({n}, {k}) (needs N % 64 == 0 and K % 128 == 0), and this device has no FlashInfer MXFP8 fallback kernel.","messagePattern":"--fp8-gemm-backend=deep_gemm cannot serve MXFP8 weight shape \\((.+?), (.+?)\\) \\(needs N % 64 == 0 and K % 128 == 0\\), and this device has no FlashInfer MXFP8 fallback kernel\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/fp8.py","lineNumber":797,"sourceCode":"            scale_u8 = layer.weight_scale_inv.data\n            # block_scale_interleave may pad and/or reshape scales,\n            # so store swizzled scales separately to keep weight update working\n            copy_or_rebind_param(\n                layer,\n                \"weight_scale_inv_swizzled\",\n                block_scale_interleave(scale_u8.contiguous()).contiguous(),\n            )\n        elif backend.is_deep_gemm():\n            from sglang.srt.layers.deep_gemm_wrapper.configurer import (\n                DEEPGEMM_SCALE_UE8M0,\n            )\n\n            n, k = layer.weight.shape\n            scale_u8 = layer.weight_scale_inv.data\n            layer.weight_scale_inv_swizzled = None\n            if n % 64 != 0 or k % 128 != 0:\n                if not (is_blackwell_supported() and is_flashinfer_available()):\n                    raise RuntimeError(\n                        f\"--fp8-gemm-backend=deep_gemm cannot serve MXFP8 weight shape \"\n                        f\"({n}, {k}) (needs N % 64 == 0 and K % 128 == 0), and this \"\n                        \"device has no FlashInfer MXFP8 fallback kernel.\"\n                    )\n                from flashinfer import block_scale_interleave\n\n                copy_or_rebind_param(\n                    layer,\n                    \"weight_scale_inv_swizzled\",\n                    block_scale_interleave(scale_u8.contiguous()).contiguous(),\n                )\n            scale_fp32 = (\n                (scale_u8.contiguous().view(-1).to(torch.int32) << 23)\n                .view(torch.float32)\n                .view(n, k // 32)\n            )\n            if DEEPGEMM_SCALE_UE8M0:\n                # Pre-packed; GEMM must be called with disable_ue8m0_cast=True.","sourceCodeStart":779,"sourceCodeEnd":815,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/fp8.py#L779-L815","documentation":"DeepGEMM's MXFP8 kernels require weight shapes with N % 64 == 0 and K % 128 == 0. When the shape violates this and the GPU is not Blackwell-with-FlashInfer (which provides a fallback via block_scale_interleave), SGLang raises instead of silently running a broken GEMM.","triggerScenarios":"--fp8-gemm-backend=deep_gemm with an MXFP8 linear layer whose weight has N not multiple of 64 or K not multiple of 128, on a non-Blackwell GPU or without FlashInfer installed; triggered during _process_mxfp8_linear_weight_scale at weight load.","commonSituations":"Running MXFP8-quantized models (e.g. Llama MX checkpoints) with deep_gemm forced on pre-Blackwell hardware (H100) or in containers lacking flashinfer; small/odd vocab or intermediate dims.","solutions":["Remove --fp8-gemm-backend=deep_gemm (or set auto) so a compatible MXFP8 backend is chosen","Install FlashInfer and run on a Blackwell (SM100) GPU to enable the swizzle fallback path","Use a checkpoint variant whose weight shapes satisfy N%64==0 and K%128==0"],"exampleFix":"# before\npython -m sglang.launch_server --model ... --fp8-gemm-backend=deep_gemm\n# after\npython -m sglang.launch_server --model ...  # default auto backend selection","handlingStrategy":"fallback","validationCode":"n, k = layer_weight_shape\nif args.fp8_gemm_backend == \"deep_gemm\" and (n % 64 or k % 128):\n    assert not (n % 64 or k % 128), \"deep_gemm mxfp8 needs N%64==0 and K%128==0; drop the flag or use Blackwell+FlashInfer\"","typeGuard":"def deep_gemm_mxfp8_shape_ok(shape) -> bool:\n    n, k = shape\n    return n % 64 == 0 and k % 128 == 0","tryCatchPattern":null,"preventionTips":["Don't hard-pin --fp8-gemm-backend=deep_gemm across heterogeneous clusters","Install flashinfer so fallback paths exist","Validate weight shapes against kernel constraints at load time"],"tags":["quantization","mxfp8","deep-gemm","hardware-compatibility","gemm-backend"],"backgroundTag":"gemm-backend-shape-unsupported","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}