sgl-project/sglang · error · NotImplementedError

The W4A8Int8 Fused MoE scheme is implemented only for NPU fo

Error message

The W4A8Int8 Fused MoE scheme is implemented only for NPU for now.

What it means

get_moe_scheme detected a dynamic per-token W4A8 INT8 scheme on MoE layers (4-bit weights, dynamic 8-bit activations). Only the NPU implementation (NPUCompressedTensorsW4A8Int8DynamicMoE) exists, so on any other device NotImplementedError is raised.

Source

Thrown at python/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py:897

                logger.info_once("Using NPUCompressedTensorsW8A8Int8DynamicMoE")
                return NPUCompressedTensorsW8A8Int8DynamicMoE(weight_quant, input_quant)
            else:
                raise NotImplementedError(
                    "The W8A8Int8 Fused MoE scheme is implemented only for NPU for now."
                )
        elif self._is_wint4afp8(weight_quant, input_quant):
            # On NPU prefer the dedicated NPU W4A8Int8 path when activations are INT8.
            if _is_npu and self._is_dynamic_token_w4a8(weight_quant, input_quant):
                logger.info_once("Using NPUCompressedTensorsW4A8Int8DynamicMoE")
                return NPUCompressedTensorsW4A8Int8DynamicMoE(self)
            logger.info_once("Using CompressedTensorsW4AFP8MoE")
            return CompressedTensorsW4AFP8MoE(self, weight_quant, input_quant)
        elif self._is_dynamic_token_w4a8(weight_quant, input_quant):
            if _is_npu:
                logger.info_once("Using NPUCompressedTensorsW4A8Int8DynamicMoE")
                return NPUCompressedTensorsW4A8Int8DynamicMoE(self)
            else:
                raise NotImplementedError(
                    "The W4A8Int8 Fused MoE scheme is implemented only for NPU for now."
                )
        else:
            raise RuntimeError(
                f"Unsupported FusedMoe scheme: {weight_quant}, {input_quant}"
            )

    def get_linear_scheme(
        self,
        layer: torch.nn.Module,
        layer_name: Optional[str] = None,
        matched_target: Optional[str] = None,
    ) -> Optional[CompressedTensorsLinearScheme]:
        """
        compressed-tensors supports non uniform in the following way:

        targets of config_groups: There can be N config_groups which each
            have a quantization scheme. Each config_group has a list of targets

View on GitHub (pinned to 0132848349)

Solutions

Re-quantize to a GPU-supported MoE scheme (W4A16, WNA16, or FP8)
Deploy on Ascend NPU hardware
Check for a W4AFP8 variant path if activations are FP8 instead of INT8 (the _is_wint4afp8 branch runs first)

Defensive patterns

Strategy: validation

Validate before calling

_is_npu = hasattr(torch, "npu") and torch.npu.is_available()
cfg = model_cfg["quantization_config"]["config"]
w, a = cfg.get("weights", {}), cfg.get("input_activations", {})
w4a8_dyn = w.get("num_bits") == 4 and a.get("num_bits") == 8 and a.get("strategy") == "dynamic"
if w4a8_dyn and not _is_npu:
    raise SystemExit("dynamic W4A8-Int8 MoE is NPU-only; re-quantize")

Prevention

Prefer W4AFP8 or W4A16 schemes for GPU serving
Verify input_activations dtype/strategy in the checkpoint config
Maintain a hardware/quantization compatibility matrix for your model fleet

When it happens

Trigger: Loading a compressed-tensors MoE checkpoint with dynamic W4A8-Int8 quantization on non-NPU hardware; _is_dynamic_token_w4a8(weight_quant, input_quant) is true but _is_npu is false.

Common situations: Serving an NPU-targeted W4A8-dynamic MoE model on NVIDIA GPUs; hardware migration without re-quantization.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/957ce0d6994a95f5. Report an issue: GitHub.