{"record":{"id":"5f16a94740f6bf7e","repo":"sgl-project/sglang","slug":"online-mxfp4-quantization-for-moe-layers-requires","errorCode":null,"errorMessage":"Online MXFP4 quantization for MoE layers requires an AMD ROCm device with FP4 hardware support (gfx95x, e.g. MI355x).","messagePattern":"Online MXFP4 quantization for MoE layers requires an AMD ROCm device with FP4 hardware support \\(gfx95x, e\\.g\\. MI355x\\)\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py","lineNumber":87,"sourceCode":"        self.input_quant = input_config\n        self.is_checkpoint_mxfp4_serialized = is_checkpoint_mxfp4_serialized\n        self.dequantization_config = dequantization_config\n\n        weight_qscheme = self.weight_quant.get(\"qscheme\")\n        input_qscheme = self.input_quant.get(\"qscheme\")\n        if not (weight_qscheme == \"per_group\" and input_qscheme == \"per_group\"):\n            raise ValueError(\n                \"For MX(FP4) Fused MoE layers, only per-group scales \"\n                \"for weights and activations are supported. Found \"\n                f\"{weight_qscheme}, {input_qscheme}\"\n            )  # noqa E501\n\n        self.static_input_scales = not self.input_quant.get(\"is_dynamic\")\n        self.with_bias = False\n\n        if not self.is_checkpoint_mxfp4_serialized:\n            if not is_gfx95_supported():\n                raise NotImplementedError(\n                    \"Online MXFP4 quantization for MoE layers requires an AMD ROCm \"\n                    \"device with FP4 hardware support (gfx95x, e.g. MI355x).\"\n                )\n            logger.info_once(\n                \"Using online MXFP4 quantization for MoE layers from a higher precision checkpoint. \"\n                \"Beware that this optimization may degrade prediction quality - please validate your model accuracy. \"\n                \"More details at https://docs.sglang.io/advanced_features/quantization.html#online-quantization.\"\n            )\n\n    @classmethod\n    def get_min_capability(cls) -> int:\n        return 70\n\n    def create_weights(\n        self,\n        layer: torch.nn.Module,\n        num_experts: int,\n        hidden_size: int,","sourceCodeStart":69,"sourceCodeEnd":105,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py#L69-L105","documentation":"Raised when QuarkW4A4MXFp4MoE is constructed with a non-MXFP4-serialized checkpoint (online quantization path) but the runtime is not an AMD gfx95x (FP4-capable, e.g. MI355x) ROCm device. Online MXFP4 quantization relies on AITER/ROCm FP4 hardware, so it cannot run on NVIDIA or older AMD GPUs.","triggerScenarios":"Loading a Quark W4A4 MoE checkpoint stored in higher precision (not pre-serialized as MXFP4) on any non-gfx95x device; is_checkpoint_mxfp4_serialized=False and is_gfx95_supported() returns False.","commonSituations":"Running an AMD-quantized Quark model on NVIDIA hardware; ROCm on older AMD gfx arch (gfx90a/gfx94x); CI machines without MI355x.","solutions":["Use a checkpoint already serialized in MXFP4 (is_checkpoint_mxfp4_serialized=True)","Run on an AMD gfx95x GPU (MI355x) with ROCm and AITER installed","Use a different quantization config (e.g. FP8) supported on your hardware"],"exampleFix":"// before\nquark_config = QuarkW4A4MXFp4Config(...)  # bf16 ckpt on H100\n// after\n# pre-quantize checkpoint to MXFP4 offline, or run on MI355x:\nquark_config = QuarkW4A4MXFp4Config(is_checkpoint_mxfp4_serialized=True, ...)","handlingStrategy":"validation","validationCode":"import torch\nassert torch.version.hip and 'gfx95' in torch.cuda.get_device_capability_name(0) if torch.cuda.is_available() else False, 'Online MXFP4 MoE needs gfx95x'","typeGuard":null,"tryCatchPattern":"try:\n    scheme = QuarkW4A4MXFp4MoE(...)\nexcept NotImplementedError as e:\n    scheme = Fp8MoEMethod(...)  # fallback scheme","preventionTips":["Pin deployment to MI355x-class GPUs for online MXFP4","Pre-quantize checkpoints to MXFP4 to avoid the online path","Gate quant config selection on is_gfx95_supported()"],"tags":["quantization","mxfp4","amd","rocm","moe","hardware-unsupported"],"backgroundTag":"unsupported-gpu-architecture","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}