sgl-project/sglang · error · RuntimeError

MXFP8 MoE quantization requires SM100 or ROCm gfx95 (gfx942

Error message

MXFP8 MoE quantization requires SM100 or ROCm gfx95 (gfx942 converts MXFP8 to block-fp8 at load instead).

What it means

MXFP8 MoE weight processing (swizzle + cutlass es kernels) is only implemented for CUDA SM100 (Blackwell) or ROCm gfx95 (MI350). On gfx942 (MI300X), MXFP8 MoE is instead converted to block-FP8 at load, so reaching this code means a routing/config bug put MXFP8 MoE on unsupported hardware.

Source

Thrown at python/sglang/srt/layers/quantization/fp8.py:1758

                qw[e] = qe
                scale[e] = se
            return qw, scale

        w13_q, w13_s = convert(layer.w13_weight.data, layer.w13_weight_scale_inv.data)
        w2_q, w2_s = convert(layer.w2_weight.data, layer.w2_weight_scale_inv.data)
        layer.w13_weight = Parameter(w13_q, requires_grad=False)
        layer.w2_weight = Parameter(w2_q, requires_grad=False)
        layer.w13_weight_scale_inv = Parameter(w13_s, requires_grad=False)
        layer.w2_weight_scale_inv = Parameter(w2_s, requires_grad=False)
        layer.w13_input_scale = None
        layer.w2_input_scale = None

    def _process_mxfp8_moe_weights(self, layer: Module, quantize: bool = True) -> None:

        if not (
            (_is_cuda and is_sm100_supported()) or (_is_hip and _is_gfx95_supported)
        ):
            raise RuntimeError(
                "MXFP8 MoE quantization requires SM100 or ROCm gfx95 "
                "(gfx942 converts MXFP8 to block-fp8 at load instead)."
            )

        def _quantize_and_swizzle_with_cutlass_es_kernel(weight: torch.Tensor):
            from sgl_kernel import es_sm100_mxfp8_blockscaled_grouped_quant

            weight = weight.contiguous()
            num_experts, m, k = weight.shape
            assert k % 32 == 0, f"{k=} must be divisible by 32 for MXFP8"

            weight_flat = weight.view(-1, k).contiguous()
            problem_sizes = torch.empty(
                (num_experts, 3), dtype=torch.int32, device=weight.device
            )
            problem_sizes[:, 0] = m
            problem_sizes[:, 1] = 0
            problem_sizes[:, 2] = k

View on GitHub (pinned to 0132848349)

Solutions

1. Run on SM100 (B200/GB200) or ROCm gfx95 (MI350) hardware for native MXFP8 MoE
2. On MI300X, let SGLang take the MXFP8→block-FP8 conversion path (update SGLang if it's not triggering)
3. Re-quantize or use an FP8 block-quant checkpoint for the available hardware

Example fix

# before: MI300X + forced mxfp8 moe path -> RuntimeError
# after
python -m sglang.launch_server --model <mxfp8-model>  # updated SGLang converts to block-fp8 on gfx942

from sglang.srt.utils import is_sm100_supported
import torch
if not ((torch.cuda.is_available() and is_sm100_supported()) or use_gfx95):
    raise SystemExit("native MXFP8 MoE needs SM100/gfx95; use conversion path or FP8 checkpoint")

def supports_native_mxfp8_moe() -> bool:
    # see fp8.py _is_cuda/_is_hip flags
    return (torch.cuda.is_available() and torch.cuda.get_device_capability(0)[0] >= 10) or is_gfx95()

When it happens

Trigger: process_weights_after_loading_block_quant dispatching to _process_mxfp8_moe_weights on a non-SM100 CUDA GPU or non-gfx95 ROCm GPU — e.g. forcing MXFP8 MoE quant on H100 or MI300X where the conversion path wasn't selected.

Common situations: Running MXFP8 MoE checkpoints on Hopper or MI300X with custom flags that bypass the gfx942 conversion path; version changes altering backend routing.

Related errors

• native MXFP8 MoE only supports gated swiglu-oai, got {activa
• --fp8-gemm-backend=deep_gemm cannot serve MXFP8 weight shape
• MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer
• MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer
• Online MXFP4 quantization for MoE layers requires an AMD ROC

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/21501e3271209408. Report an issue: GitHub.