sgl-project/sglang · error · ValueError

For INT8 Fused MoE layers, we require channelwise, dynamic p

Error message

For INT8 Fused MoE layers, we require channelwise, dynamic per token quantization. Found {self.weight_quant}, {self.input_quant}

What it means

INT8 Fused MoE requires exactly channelwise weight quantization combined with dynamic per-token input quantization. This error fires when weight strategy != CHANNEL or input strategy != TOKEN, printing both configs to show the mismatch.

Source

Thrown at python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8_moe.py:44

logger = logging.getLogger(__name__)


class NPUCompressedTensorsW8A8Int8DynamicMoE(CompressedTensorsMoEScheme):

    def __init__(self, weight_quant, input_quant):
        self.weight_quant = weight_quant
        self.input_quant = input_quant
        self.w13_kernel = NPUW8A8Int8MoEMethod()
        self.w2_kernel = NPUW8A8Int8MoEMethod()

        self.static_input_scales = not self.input_quant.dynamic
        per_channel = (
            self.weight_quant.strategy == QuantizationStrategy.CHANNEL
            and self.input_quant.strategy == QuantizationStrategy.TOKEN
        )
        if not per_channel:
            raise ValueError(
                "For INT8 Fused MoE layers, we require channelwise, "
                "dynamic per token quantization. Found "
                f"{self.weight_quant}, {self.input_quant}"
            )

        self.static_input_scales = not self.input_quant.dynamic
        if self.static_input_scales:
            raise ValueError(
                "For INT8 Fused MoE layers, we require channelwise, "
                "dynamic per token quantization. Found static input scales."
            )

    def create_weights(
        self,
        layer: torch.nn.Module,
        num_experts: int,
        hidden_size: int,
        intermediate_size_per_partition: int,

View on GitHub (pinned to 0132848349)

Solutions

1. Re-quantize with weights strategy=channel and input strategy=token, dynamic=true
2. Use the FP8 MoE path if the recipe can't be changed
3. Check the printed weight_quant/input_quant in the error output against the requirement

Example fix

// before
"weights": {"strategy": "tensor"}
// after
"weights": {"strategy": "channel"}, "input_quant": {"strategy": "token", "dynamic": true}

w, iq = cfg["weights"], cfg["input_quant"]
assert w["strategy"] == "channel" and iq["strategy"] == "token", "INT8 MoE needs channel weights + token activations"

When it happens

Trigger: Loading a compressed-tensors INT8 MoE model whose weights are per-tensor or grouped, or whose activations are per-tensor, in CompressedTensorsW8A8Int8MoE.__init__.

Common situations: Applying a generic INT8 recipe (e.g. per-tensor weights) to a MoE model; recipes tuned for linear layers reused on MoE.

Related errors

• The W8A8Int8 Fused MoE scheme is implemented only for NPU fo
• Unsupported weight quantization strategy: {self.weight_quant
• Unknown quantization strategy {self.strategy}
• For INT8 Fused MoE layers, we require channelwise, dynamic p
• Comfy INT8 embedding weights support lookup only

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/982ef3069d7c73c2. Report an issue: GitHub.