sgl-project/sglang · error · ValueError

For INT8 Fused MoE layers, we require channelwise, dynamic…

Error message

For INT8 Fused MoE layers, we require channelwise, dynamic per token quantization. Found static input scales.

What it means

Second gate for INT8 MoE: even with correct channel/token strategies, static input scales are rejected — input_quant.dynamic must be true. Static calibrated activation scales are unsupported for INT8 fused MoE kernels.

Solutions

Re-quantize with dynamic activation quantization (drop input_scale tensors, set dynamic=true)
Switch to the FP8 MoE scheme if static scales are mandatory

Example fix

// before
"input_quant": {"strategy": "token", "dynamic": false}
// after
"input_quant": {"strategy": "token", "dynamic": true}

Defensive patterns

Strategy: validation

Validate before calling

assert cfg["input_quant"].get("dynamic", True), "INT8 MoE rejects static input scales"

Prevention

Skip calibration scale export for INT8 MoE models

When it happens

Trigger: input_quant.dynamic == False on a compressed-tensors INT8 MoE checkpoint, i.e. calibration scales embedded in the model.

Common situations: Loading a statically-calibrated INT8 MoE export; conversion tools defaulting to static activation quantization.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/985e004351f38ac1. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8_moe.py:52

        self.input_quant = input_quant
        self.w13_kernel = NPUW8A8Int8MoEMethod()
        self.w2_kernel = NPUW8A8Int8MoEMethod()

        self.static_input_scales = not self.input_quant.dynamic
        per_channel = (
            self.weight_quant.strategy == QuantizationStrategy.CHANNEL
            and self.input_quant.strategy == QuantizationStrategy.TOKEN
        )
        if not per_channel:
            raise ValueError(
                "For INT8 Fused MoE layers, we require channelwise, "
                "dynamic per token quantization. Found "
                f"{self.weight_quant}, {self.input_quant}"
            )

        self.static_input_scales = not self.input_quant.dynamic
        if self.static_input_scales:
            raise ValueError(
                "For INT8 Fused MoE layers, we require channelwise, "
                "dynamic per token quantization. Found static input scales."
            )

    def create_weights(
        self,
        layer: torch.nn.Module,
        num_experts: int,
        hidden_size: int,
        intermediate_size_per_partition: int,
        params_dtype: torch.dtype,
        **extra_weight_attrs,
    ):

        from sglang.srt.layers.moe.fused_moe_triton import FusedMoeWeightScaleSupported

        params_dtype = torch.int8

View on GitHub (pinned to 0132848349)