sgl-project/sglang · error · NotImplementedError

The W8A8Int8 Fused MoE scheme is implemented only for NPU fo

Error message

The W8A8Int8 Fused MoE scheme is implemented only for NPU for now.

What it means

get_moe_scheme detected a dynamic per-token W8A8 INT8 quantized MoE layer (dynamic weight+activation int8 scheme). The NPUCompressedTensorsW8A8Int8DynamicMoE implementation exists only for Ascend NPUs, so on CUDA/other devices the scheme is refused with NotImplementedError.

Source

Thrown at python/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py:882

            else:
                if (
                    self._is_dynamic_token_w4(weight_quant, input_quant)
                    and input_quant is None
                ):
                    logger.info_once("Using NPUCompressedTensorsW4A16Int4DynamicMoE")
                    return NPUCompressedTensorsW4A16Int4DynamicMoE(self)
        elif self._is_fp4a4_nvfp4(weight_quant, input_quant):
            logger.info_once("Using CompressedTensorsW4A4Nvfp4MoE")
            return CompressedTensorsW4A4Nvfp4MoE()
        elif self._is_fp8_w8a8(weight_quant, input_quant):
            logger.info_once("Using CompressedTensorsW8A8Fp8MoE")
            return CompressedTensorsW8A8Fp8MoE(weight_quant, input_quant)
        elif self._is_dynamic_token_w8a8(weight_quant, input_quant):
            if _is_npu:
                logger.info_once("Using NPUCompressedTensorsW8A8Int8DynamicMoE")
                return NPUCompressedTensorsW8A8Int8DynamicMoE(weight_quant, input_quant)
            else:
                raise NotImplementedError(
                    "The W8A8Int8 Fused MoE scheme is implemented only for NPU for now."
                )
        elif self._is_wint4afp8(weight_quant, input_quant):
            # On NPU prefer the dedicated NPU W4A8Int8 path when activations are INT8.
            if _is_npu and self._is_dynamic_token_w4a8(weight_quant, input_quant):
                logger.info_once("Using NPUCompressedTensorsW4A8Int8DynamicMoE")
                return NPUCompressedTensorsW4A8Int8DynamicMoE(self)
            logger.info_once("Using CompressedTensorsW4AFP8MoE")
            return CompressedTensorsW4AFP8MoE(self, weight_quant, input_quant)
        elif self._is_dynamic_token_w4a8(weight_quant, input_quant):
            if _is_npu:
                logger.info_once("Using NPUCompressedTensorsW4A8Int8DynamicMoE")
                return NPUCompressedTensorsW4A8Int8DynamicMoE(self)
            else:
                raise NotImplementedError(
                    "The W4A8Int8 Fused MoE scheme is implemented only for NPU for now."
                )
        else:

View on GitHub (pinned to 0132848349)

Solutions

Use a non-dynamic W8A8 scheme or an FP8-quantized checkpoint for GPU serving
Run on Ascend NPU hardware where the NPU dynamic W8A8 path is implemented
Re-quantize the model with llmcompressor to a GPU-supported scheme (static W8A8Int8 or FP8)

Example fix

# before: checkpoint quantized dynamic W8A8-Int8
# after: re-quantize with llmcompressor
from llmcompressor.transformers import oneshot
recipe = Int8WeightOnlyModifier(group_size=-1, targets="Linear", ignore="lm_head")
# or use FP8Modifier for GPU

Defensive patterns

Strategy: validation

Validate before calling

import torch
_is_npu = hasattr(torch, "npu") and torch.npu.is_available()
w = model_cfg["quantization_config"]["config"].get("weights", {})
dynamic_int8 = (w.get("num_bits") == 8 and w.get("strategy") == "dynamic")
if dynamic_int8 and not _is_npu:
    raise SystemExit("dynamic W8A8-Int8 MoE requires NPU; re-quantize for GPU")

Prevention

Match checkpoint quantization family to deployment hardware
Keep GPU-targeted (FP8/static W8A8) and NPU-targeted checkpoints separate
Validate quantization_config at model-download time, not launch time

When it happens

Trigger: Loading a compressed-tensors checkpoint with dynamic per-token W8A8 INT8 quantization on the MoE (expert) layers on non-NPU hardware, e.g. a llmcompressor W8A8-Dynamic model on a GPU server.

Common situations: Running an Intel/NPU-targeted quantized MoE checkpoint on NVIDIA GPUs; mixing hardware-specific checkpoints across platforms.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/683ad853e6dbc005. Report an issue: GitHub.