sgl-project/sglang · error · ValueError

For INT8 Fused MoE layers, we require channelwise, dynamic p

Error message

For INT8 Fused MoE layers, we require channelwise, dynamic per token quantization. Found static input scales.

What it means

Second gate for INT8 MoE: even with correct channel/token strategies, static input scales are rejected — input_quant.dynamic must be true. Static calibrated activation scales are unsupported for INT8 fused MoE kernels.

Source

Thrown at python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8_moe.py:52

        self.input_quant = input_quant
        self.w13_kernel = NPUW8A8Int8MoEMethod()
        self.w2_kernel = NPUW8A8Int8MoEMethod()

        self.static_input_scales = not self.input_quant.dynamic
        per_channel = (
            self.weight_quant.strategy == QuantizationStrategy.CHANNEL
            and self.input_quant.strategy == QuantizationStrategy.TOKEN
        )
        if not per_channel:
            raise ValueError(
                "For INT8 Fused MoE layers, we require channelwise, "
                "dynamic per token quantization. Found "
                f"{self.weight_quant}, {self.input_quant}"
            )

        self.static_input_scales = not self.input_quant.dynamic
        if self.static_input_scales:
            raise ValueError(
                "For INT8 Fused MoE layers, we require channelwise, "
                "dynamic per token quantization. Found static input scales."
            )

    def create_weights(
        self,
        layer: torch.nn.Module,
        num_experts: int,
        hidden_size: int,
        intermediate_size_per_partition: int,
        params_dtype: torch.dtype,
        **extra_weight_attrs,
    ):

        from sglang.srt.layers.moe.fused_moe_triton import FusedMoeWeightScaleSupported

        params_dtype = torch.int8

View on GitHub (pinned to 0132848349)

Solutions

1. Re-quantize with dynamic activation quantization (drop input_scale tensors, set dynamic=true)
2. Switch to the FP8 MoE scheme if static scales are mandatory

Example fix

// before
"input_quant": {"strategy": "token", "dynamic": false}
// after
"input_quant": {"strategy": "token", "dynamic": true}

assert cfg["input_quant"].get("dynamic", True), "INT8 MoE rejects static input scales"

When it happens

Trigger: input_quant.dynamic == False on a compressed-tensors INT8 MoE checkpoint, i.e. calibration scales embedded in the model.

Common situations: Loading a statically-calibrated INT8 MoE export; conversion tools defaulting to static activation quantization.

Related errors

• The W8A8Int8 Fused MoE scheme is implemented only for NPU fo
• For FP8 Fused MoE layer, we require either per tensor or cha
• Static compressed-tensors scheme is not yet supported on NPU
• For INT8 Fused MoE layers, we require channelwise, dynamic p
• Comfy INT8 embedding weights support lookup only

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/985e004351f38ac1. Report an issue: GitHub.