{"record":{"id":"d2aee81a73b91c3b","repo":"sgl-project/sglang","slug":"online-mxfp4-quantization-for-moe-is-only-supporte","errorCode":null,"errorMessage":"Online MXFP4 quantization for MoE is only supported on AMD GPUs.","messagePattern":"Online MXFP4 quantization for MoE is only supported on AMD GPUs\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py","lineNumber":528,"sourceCode":"            layer,\n            f\"{prefix}_weight_scale\",\n            torch.nn.Parameter(mxfp4_scale, requires_grad=False),\n        )\n\n    def get_online_weight_loader(self, layer, original_weight_loader):\n        \"\"\"\n        Wrap the original weight loader to perform online MXFP4 quantization for MoE layers.\n        \"\"\"\n\n        def online_mxfp4_moe_weight_loader(\n            param: torch.nn.Parameter,\n            loaded_weight: torch.Tensor,\n            weight_name: str,\n            shard_id: str,\n            expert_id: int,\n        ):\n            if dynamic_mxfp4_quant is None:\n                raise NotImplementedError(\n                    \"Online MXFP4 quantization for MoE is only supported on AMD GPUs.\"\n                )\n\n            # Determine which weight parameter we're loading (w13 or w2)\n            is_w13 = \"w13\" in weight_name\n            is_w2 = \"w2\" in weight_name\n\n            # Initialize weight on device if first load\n            if is_w13 and layer._w13_loaded_numel == 0:\n                layer.w13_weight = torch.nn.Parameter(\n                    torch.empty_like(param.data, device=layer._load_device),\n                    requires_grad=False,\n                )\n                param = layer.w13_weight\n            elif is_w2 and layer._w2_loaded_numel == 0:\n                layer.w2_weight = torch.nn.Parameter(\n                    torch.empty_like(param.data, device=layer._load_device),\n                    requires_grad=False,","sourceCodeStart":510,"sourceCodeEnd":546,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py#L510-L546","documentation":"online_mxfp4_moe_weight_loader needs AITER's dynamic_mxfp4_quant (AMD ROCm). If dynamic_mxfp4_quant is None (AITER import failed or non-ROCm platform), the stub raises NotImplementedError when loading higher-precision MoE weights for online quantization.","triggerScenarios":"Online quantization path reached on NVIDIA GPUs or ROCm without AITER installed; dynamic_mxfp4_quant is the ImportError stub from quark/utils.py.","commonSituations":"Missing/pip-broken aiter package on MI300x/MI355x; accidentally running online MXFP4 path on CUDA; aiter built for wrong ROCm version.","solutions":["Install/repair AITER matching your ROCm version (pip install aiter) on the AMD GPU host","Use a pre-quantized MXFP4 checkpoint so online quantization isn't needed","Verify torch.version.hip is set and the correct backend was selected"],"exampleFix":"# before: aiter missing -> NotImplementedError at load time\n# after\npip install aiter --index-url https://rocm.nightlies.amd.com/v2/gfx95-docker/","handlingStrategy":"validation","validationCode":"from sglang.srt.layers.quantization.quark.utils import dynamic_mxfp4_quant\nimport torch\nassert dynamic_mxfp4_quant is not None and torch.version.hip, 'AITER/ROCm required for online MXFP4'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Smoke-test `import aiter` before server launch","Use pre-quantized MXFP4 checkpoints on uncertain environments"],"tags":["quantization","mxfp4","aiter","rocm","dependency-missing"],"backgroundTag":"missing-native-dependency","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}