{"record":{"id":"683ad853e6dbc005","repo":"sgl-project/sglang","slug":"the-w8a8int8-fused-moe-scheme-is-implemented-only","errorCode":null,"errorMessage":"The W8A8Int8 Fused MoE scheme is implemented only for NPU for now.","messagePattern":"The W8A8Int8 Fused MoE scheme is implemented only for NPU for now\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py","lineNumber":882,"sourceCode":"            else:\n                if (\n                    self._is_dynamic_token_w4(weight_quant, input_quant)\n                    and input_quant is None\n                ):\n                    logger.info_once(\"Using NPUCompressedTensorsW4A16Int4DynamicMoE\")\n                    return NPUCompressedTensorsW4A16Int4DynamicMoE(self)\n        elif self._is_fp4a4_nvfp4(weight_quant, input_quant):\n            logger.info_once(\"Using CompressedTensorsW4A4Nvfp4MoE\")\n            return CompressedTensorsW4A4Nvfp4MoE()\n        elif self._is_fp8_w8a8(weight_quant, input_quant):\n            logger.info_once(\"Using CompressedTensorsW8A8Fp8MoE\")\n            return CompressedTensorsW8A8Fp8MoE(weight_quant, input_quant)\n        elif self._is_dynamic_token_w8a8(weight_quant, input_quant):\n            if _is_npu:\n                logger.info_once(\"Using NPUCompressedTensorsW8A8Int8DynamicMoE\")\n                return NPUCompressedTensorsW8A8Int8DynamicMoE(weight_quant, input_quant)\n            else:\n                raise NotImplementedError(\n                    \"The W8A8Int8 Fused MoE scheme is implemented only for NPU for now.\"\n                )\n        elif self._is_wint4afp8(weight_quant, input_quant):\n            # On NPU prefer the dedicated NPU W4A8Int8 path when activations are INT8.\n            if _is_npu and self._is_dynamic_token_w4a8(weight_quant, input_quant):\n                logger.info_once(\"Using NPUCompressedTensorsW4A8Int8DynamicMoE\")\n                return NPUCompressedTensorsW4A8Int8DynamicMoE(self)\n            logger.info_once(\"Using CompressedTensorsW4AFP8MoE\")\n            return CompressedTensorsW4AFP8MoE(self, weight_quant, input_quant)\n        elif self._is_dynamic_token_w4a8(weight_quant, input_quant):\n            if _is_npu:\n                logger.info_once(\"Using NPUCompressedTensorsW4A8Int8DynamicMoE\")\n                return NPUCompressedTensorsW4A8Int8DynamicMoE(self)\n            else:\n                raise NotImplementedError(\n                    \"The W4A8Int8 Fused MoE scheme is implemented only for NPU for now.\"\n                )\n        else:","sourceCodeStart":864,"sourceCodeEnd":900,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py#L864-L900","documentation":"get_moe_scheme detected a dynamic per-token W8A8 INT8 quantized MoE layer (dynamic weight+activation int8 scheme). The NPUCompressedTensorsW8A8Int8DynamicMoE implementation exists only for Ascend NPUs, so on CUDA/other devices the scheme is refused with NotImplementedError.","triggerScenarios":"Loading a compressed-tensors checkpoint with dynamic per-token W8A8 INT8 quantization on the MoE (expert) layers on non-NPU hardware, e.g. a llmcompressor W8A8-Dynamic model on a GPU server.","commonSituations":"Running an Intel/NPU-targeted quantized MoE checkpoint on NVIDIA GPUs; mixing hardware-specific checkpoints across platforms.","solutions":["Use a non-dynamic W8A8 scheme or an FP8-quantized checkpoint for GPU serving","Run on Ascend NPU hardware where the NPU dynamic W8A8 path is implemented","Re-quantize the model with llmcompressor to a GPU-supported scheme (static W8A8Int8 or FP8)"],"exampleFix":"# before: checkpoint quantized dynamic W8A8-Int8\n# after: re-quantize with llmcompressor\nfrom llmcompressor.transformers import oneshot\nrecipe = Int8WeightOnlyModifier(group_size=-1, targets=\"Linear\", ignore=\"lm_head\")\n# or use FP8Modifier for GPU","handlingStrategy":"validation","validationCode":"import torch\n_is_npu = hasattr(torch, \"npu\") and torch.npu.is_available()\nw = model_cfg[\"quantization_config\"][\"config\"].get(\"weights\", {})\ndynamic_int8 = (w.get(\"num_bits\") == 8 and w.get(\"strategy\") == \"dynamic\")\nif dynamic_int8 and not _is_npu:\n    raise SystemExit(\"dynamic W8A8-Int8 MoE requires NPU; re-quantize for GPU\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Match checkpoint quantization family to deployment hardware","Keep GPU-targeted (FP8/static W8A8) and NPU-targeted checkpoints separate","Validate quantization_config at model-download time, not launch time"],"tags":["quantization","moe","int8","npu","hardware-support"],"backgroundTag":"hardware-unsupported-quantization-scheme","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}