{"record":{"id":"957ce0d6994a95f5","repo":"sgl-project/sglang","slug":"the-w4a8int8-fused-moe-scheme-is-implemented-only","errorCode":null,"errorMessage":"The W4A8Int8 Fused MoE scheme is implemented only for NPU for now.","messagePattern":"The W4A8Int8 Fused MoE scheme is implemented only for NPU for now\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py","lineNumber":897,"sourceCode":"                logger.info_once(\"Using NPUCompressedTensorsW8A8Int8DynamicMoE\")\n                return NPUCompressedTensorsW8A8Int8DynamicMoE(weight_quant, input_quant)\n            else:\n                raise NotImplementedError(\n                    \"The W8A8Int8 Fused MoE scheme is implemented only for NPU for now.\"\n                )\n        elif self._is_wint4afp8(weight_quant, input_quant):\n            # On NPU prefer the dedicated NPU W4A8Int8 path when activations are INT8.\n            if _is_npu and self._is_dynamic_token_w4a8(weight_quant, input_quant):\n                logger.info_once(\"Using NPUCompressedTensorsW4A8Int8DynamicMoE\")\n                return NPUCompressedTensorsW4A8Int8DynamicMoE(self)\n            logger.info_once(\"Using CompressedTensorsW4AFP8MoE\")\n            return CompressedTensorsW4AFP8MoE(self, weight_quant, input_quant)\n        elif self._is_dynamic_token_w4a8(weight_quant, input_quant):\n            if _is_npu:\n                logger.info_once(\"Using NPUCompressedTensorsW4A8Int8DynamicMoE\")\n                return NPUCompressedTensorsW4A8Int8DynamicMoE(self)\n            else:\n                raise NotImplementedError(\n                    \"The W4A8Int8 Fused MoE scheme is implemented only for NPU for now.\"\n                )\n        else:\n            raise RuntimeError(\n                f\"Unsupported FusedMoe scheme: {weight_quant}, {input_quant}\"\n            )\n\n    def get_linear_scheme(\n        self,\n        layer: torch.nn.Module,\n        layer_name: Optional[str] = None,\n        matched_target: Optional[str] = None,\n    ) -> Optional[CompressedTensorsLinearScheme]:\n        \"\"\"\n        compressed-tensors supports non uniform in the following way:\n\n        targets of config_groups: There can be N config_groups which each\n            have a quantization scheme. Each config_group has a list of targets","sourceCodeStart":879,"sourceCodeEnd":915,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py#L879-L915","documentation":"get_moe_scheme detected a dynamic per-token W4A8 INT8 scheme on MoE layers (4-bit weights, dynamic 8-bit activations). Only the NPU implementation (NPUCompressedTensorsW4A8Int8DynamicMoE) exists, so on any other device NotImplementedError is raised.","triggerScenarios":"Loading a compressed-tensors MoE checkpoint with dynamic W4A8-Int8 quantization on non-NPU hardware; _is_dynamic_token_w4a8(weight_quant, input_quant) is true but _is_npu is false.","commonSituations":"Serving an NPU-targeted W4A8-dynamic MoE model on NVIDIA GPUs; hardware migration without re-quantization.","solutions":["Re-quantize to a GPU-supported MoE scheme (W4A16, WNA16, or FP8)","Deploy on Ascend NPU hardware","Check for a W4AFP8 variant path if activations are FP8 instead of INT8 (the _is_wint4afp8 branch runs first)"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"_is_npu = hasattr(torch, \"npu\") and torch.npu.is_available()\ncfg = model_cfg[\"quantization_config\"][\"config\"]\nw, a = cfg.get(\"weights\", {}), cfg.get(\"input_activations\", {})\nw4a8_dyn = w.get(\"num_bits\") == 4 and a.get(\"num_bits\") == 8 and a.get(\"strategy\") == \"dynamic\"\nif w4a8_dyn and not _is_npu:\n    raise SystemExit(\"dynamic W4A8-Int8 MoE is NPU-only; re-quantize\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Prefer W4AFP8 or W4A16 schemes for GPU serving","Verify input_activations dtype/strategy in the checkpoint config","Maintain a hardware/quantization compatibility matrix for your model fleet"],"tags":["quantization","moe","w4a8","npu","hardware-support"],"backgroundTag":"hardware-unsupported-quantization-scheme","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}