sgl-project/sglang · critical · NotImplementedError

Online MXFP4 quantization for MoE layers requires an AMD ROC

Error message

Online MXFP4 quantization for MoE layers requires an AMD ROCm device with FP4 hardware support (gfx95x, e.g. MI355x).

What it means

Raised when QuarkW4A4MXFp4MoE is constructed with a non-MXFP4-serialized checkpoint (online quantization path) but the runtime is not an AMD gfx95x (FP4-capable, e.g. MI355x) ROCm device. Online MXFP4 quantization relies on AITER/ROCm FP4 hardware, so it cannot run on NVIDIA or older AMD GPUs.

Source

Thrown at python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py:87

        self.input_quant = input_config
        self.is_checkpoint_mxfp4_serialized = is_checkpoint_mxfp4_serialized
        self.dequantization_config = dequantization_config

        weight_qscheme = self.weight_quant.get("qscheme")
        input_qscheme = self.input_quant.get("qscheme")
        if not (weight_qscheme == "per_group" and input_qscheme == "per_group"):
            raise ValueError(
                "For MX(FP4) Fused MoE layers, only per-group scales "
                "for weights and activations are supported. Found "
                f"{weight_qscheme}, {input_qscheme}"
            )  # noqa E501

        self.static_input_scales = not self.input_quant.get("is_dynamic")
        self.with_bias = False

        if not self.is_checkpoint_mxfp4_serialized:
            if not is_gfx95_supported():
                raise NotImplementedError(
                    "Online MXFP4 quantization for MoE layers requires an AMD ROCm "
                    "device with FP4 hardware support (gfx95x, e.g. MI355x)."
                )
            logger.info_once(
                "Using online MXFP4 quantization for MoE layers from a higher precision checkpoint. "
                "Beware that this optimization may degrade prediction quality - please validate your model accuracy. "
                "More details at https://docs.sglang.io/advanced_features/quantization.html#online-quantization."
            )

    @classmethod
    def get_min_capability(cls) -> int:
        return 70

    def create_weights(
        self,
        layer: torch.nn.Module,
        num_experts: int,
        hidden_size: int,

View on GitHub (pinned to 0132848349)

Solutions

Use a checkpoint already serialized in MXFP4 (is_checkpoint_mxfp4_serialized=True)
Run on an AMD gfx95x GPU (MI355x) with ROCm and AITER installed
Use a different quantization config (e.g. FP8) supported on your hardware

Example fix

// before
quark_config = QuarkW4A4MXFp4Config(...)  # bf16 ckpt on H100
// after
# pre-quantize checkpoint to MXFP4 offline, or run on MI355x:
quark_config = QuarkW4A4MXFp4Config(is_checkpoint_mxfp4_serialized=True, ...)

Defensive patterns

Strategy: validation

Validate before calling

import torch
assert torch.version.hip and 'gfx95' in torch.cuda.get_device_capability_name(0) if torch.cuda.is_available() else False, 'Online MXFP4 MoE needs gfx95x'

Try / catch

try:
    scheme = QuarkW4A4MXFp4MoE(...)
except NotImplementedError as e:
    scheme = Fp8MoEMethod(...)  # fallback scheme

Prevention

Pin deployment to MI355x-class GPUs for online MXFP4
Pre-quantize checkpoints to MXFP4 to avoid the online path
Gate quant config selection on is_gfx95_supported()

When it happens

Trigger: Loading a Quark W4A4 MoE checkpoint stored in higher precision (not pre-serialized as MXFP4) on any non-gfx95x device; is_checkpoint_mxfp4_serialized=False and is_gfx95_supported() returns False.

Common situations: Running an AMD-quantized Quark model on NVIDIA hardware; ROCm on older AMD gfx arch (gfx90a/gfx94x); CI machines without MI355x.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5f16a94740f6bf7e. Report an issue: GitHub.