{"record":{"id":"21501e3271209408","repo":"sgl-project/sglang","slug":"mxfp8-moe-quantization-requires-sm100-or-rocm-gfx9","errorCode":null,"errorMessage":"MXFP8 MoE quantization requires SM100 or ROCm gfx95 (gfx942 converts MXFP8 to block-fp8 at load instead).","messagePattern":"MXFP8 MoE quantization requires SM100 or ROCm gfx95 \\(gfx942 converts MXFP8 to block-fp8 at load instead\\)\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/fp8.py","lineNumber":1758,"sourceCode":"                qw[e] = qe\n                scale[e] = se\n            return qw, scale\n\n        w13_q, w13_s = convert(layer.w13_weight.data, layer.w13_weight_scale_inv.data)\n        w2_q, w2_s = convert(layer.w2_weight.data, layer.w2_weight_scale_inv.data)\n        layer.w13_weight = Parameter(w13_q, requires_grad=False)\n        layer.w2_weight = Parameter(w2_q, requires_grad=False)\n        layer.w13_weight_scale_inv = Parameter(w13_s, requires_grad=False)\n        layer.w2_weight_scale_inv = Parameter(w2_s, requires_grad=False)\n        layer.w13_input_scale = None\n        layer.w2_input_scale = None\n\n    def _process_mxfp8_moe_weights(self, layer: Module, quantize: bool = True) -> None:\n\n        if not (\n            (_is_cuda and is_sm100_supported()) or (_is_hip and _is_gfx95_supported)\n        ):\n            raise RuntimeError(\n                \"MXFP8 MoE quantization requires SM100 or ROCm gfx95 \"\n                \"(gfx942 converts MXFP8 to block-fp8 at load instead).\"\n            )\n\n        def _quantize_and_swizzle_with_cutlass_es_kernel(weight: torch.Tensor):\n            from sgl_kernel import es_sm100_mxfp8_blockscaled_grouped_quant\n\n            weight = weight.contiguous()\n            num_experts, m, k = weight.shape\n            assert k % 32 == 0, f\"{k=} must be divisible by 32 for MXFP8\"\n\n            weight_flat = weight.view(-1, k).contiguous()\n            problem_sizes = torch.empty(\n                (num_experts, 3), dtype=torch.int32, device=weight.device\n            )\n            problem_sizes[:, 0] = m\n            problem_sizes[:, 1] = 0\n            problem_sizes[:, 2] = k","sourceCodeStart":1740,"sourceCodeEnd":1776,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/fp8.py#L1740-L1776","documentation":"MXFP8 MoE weight processing (swizzle + cutlass es kernels) is only implemented for CUDA SM100 (Blackwell) or ROCm gfx95 (MI350). On gfx942 (MI300X), MXFP8 MoE is instead converted to block-FP8 at load, so reaching this code means a routing/config bug put MXFP8 MoE on unsupported hardware.","triggerScenarios":"process_weights_after_loading_block_quant dispatching to _process_mxfp8_moe_weights on a non-SM100 CUDA GPU or non-gfx95 ROCm GPU — e.g. forcing MXFP8 MoE quant on H100 or MI300X where the conversion path wasn't selected.","commonSituations":"Running MXFP8 MoE checkpoints on Hopper or MI300X with custom flags that bypass the gfx942 conversion path; version changes altering backend routing.","solutions":["Run on SM100 (B200/GB200) or ROCm gfx95 (MI350) hardware for native MXFP8 MoE","On MI300X, let SGLang take the MXFP8→block-FP8 conversion path (update SGLang if it's not triggering)","Re-quantize or use an FP8 block-quant checkpoint for the available hardware"],"exampleFix":"# before: MI300X + forced mxfp8 moe path -> RuntimeError\n# after\npython -m sglang.launch_server --model <mxfp8-model>  # updated SGLang converts to block-fp8 on gfx942","handlingStrategy":"validation","validationCode":"from sglang.srt.utils import is_sm100_supported\nimport torch\nif not ((torch.cuda.is_available() and is_sm100_supported()) or use_gfx95):\n    raise SystemExit(\"native MXFP8 MoE needs SM100/gfx95; use conversion path or FP8 checkpoint\")","typeGuard":"def supports_native_mxfp8_moe() -> bool:\n    # see fp8.py _is_cuda/_is_hip flags\n    return (torch.cuda.is_available() and torch.cuda.get_device_capability(0)[0] >= 10) or is_gfx95()","tryCatchPattern":null,"preventionTips":["Match quant format to hardware before launching","Keep SGLang updated so gfx942 conversion routing works"],"tags":["quantization","mxfp8","moe","hardware-compatibility","rocm"],"backgroundTag":"hardware-unsupported-operation","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}