sgl-project/sglang · critical · NotImplementedError

Online MXFP4 quantization for MoE is only supported on AMD G

Error message

Online MXFP4 quantization for MoE is only supported on AMD GPUs.

What it means

online_mxfp4_moe_weight_loader needs AITER's dynamic_mxfp4_quant (AMD ROCm). If dynamic_mxfp4_quant is None (AITER import failed or non-ROCm platform), the stub raises NotImplementedError when loading higher-precision MoE weights for online quantization.

Source

Thrown at python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py:528

            layer,
            f"{prefix}_weight_scale",
            torch.nn.Parameter(mxfp4_scale, requires_grad=False),
        )

    def get_online_weight_loader(self, layer, original_weight_loader):
        """
        Wrap the original weight loader to perform online MXFP4 quantization for MoE layers.
        """

        def online_mxfp4_moe_weight_loader(
            param: torch.nn.Parameter,
            loaded_weight: torch.Tensor,
            weight_name: str,
            shard_id: str,
            expert_id: int,
        ):
            if dynamic_mxfp4_quant is None:
                raise NotImplementedError(
                    "Online MXFP4 quantization for MoE is only supported on AMD GPUs."
                )

            # Determine which weight parameter we're loading (w13 or w2)
            is_w13 = "w13" in weight_name
            is_w2 = "w2" in weight_name

            # Initialize weight on device if first load
            if is_w13 and layer._w13_loaded_numel == 0:
                layer.w13_weight = torch.nn.Parameter(
                    torch.empty_like(param.data, device=layer._load_device),
                    requires_grad=False,
                )
                param = layer.w13_weight
            elif is_w2 and layer._w2_loaded_numel == 0:
                layer.w2_weight = torch.nn.Parameter(
                    torch.empty_like(param.data, device=layer._load_device),
                    requires_grad=False,

View on GitHub (pinned to 0132848349)

Solutions

1. Install/repair AITER matching your ROCm version (pip install aiter) on the AMD GPU host
2. Use a pre-quantized MXFP4 checkpoint so online quantization isn't needed
3. Verify torch.version.hip is set and the correct backend was selected

Example fix

# before: aiter missing -> NotImplementedError at load time
# after
pip install aiter --index-url https://rocm.nightlies.amd.com/v2/gfx95-docker/

from sglang.srt.layers.quantization.quark.utils import dynamic_mxfp4_quant
import torch
assert dynamic_mxfp4_quant is not None and torch.version.hip, 'AITER/ROCm required for online MXFP4'

When it happens

Trigger: Online quantization path reached on NVIDIA GPUs or ROCm without AITER installed; dynamic_mxfp4_quant is the ImportError stub from quark/utils.py.

Common situations: Missing/pip-broken aiter package on MI300x/MI355x; accidentally running online MXFP4 path on CUDA; aiter built for wrong ROCm version.

Related errors

• Online MXFP4 quantization for MoE layers requires an AMD ROC
• AITer backend does not have a metadata builder.
• AITer backend requires num_heads ({num_heads}) to be a multi
• AITER Sage backend does not have a metadata builder.
• AITER Sage attention is not available, please update AITER v

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d2aee81a73b91c3b. Report an issue: GitHub.