sgl-project/sglang · error · ValueError

For FP8 Fused MoE layer, we require either per tensor or cha

Error message

For FP8 Fused MoE layer, we require either per tensor or channelwise, dynamic per token quantization.

What it means

For FP8 Fused MoE, static (calibrated) input scales are only supported with per-tensor input quantization. This error fires when input scales are static (input_quant.dynamic == False) AND the quantization is per-channel/per-token — a combination the fused MoE kernels cannot consume.

Source

Thrown at python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8_moe.py:76

        per_tensor = (
            self.weight_quant.strategy == QuantizationStrategy.TENSOR
            and self.input_quant.strategy == QuantizationStrategy.TENSOR
        )
        per_channel = (
            self.weight_quant.strategy == QuantizationStrategy.CHANNEL
            and self.input_quant.strategy == QuantizationStrategy.TOKEN
        )
        if not (per_tensor or per_channel):
            assert self.weight_quant.strategy == QuantizationStrategy.BLOCK
            self.weight_block_size = self.weight_quant.block_structure
            assert self.weight_quant.dynamic is not None
        else:
            self.weight_block_size = None
        self.block_quant = self.weight_block_size is not None

        self.static_input_scales = not self.input_quant.dynamic
        if self.static_input_scales and per_channel:
            raise ValueError(
                "For FP8 Fused MoE layer, we require either per tensor or "
                "channelwise, dynamic per token quantization."
            )

    @classmethod
    def get_min_capability(cls) -> int:
        # ampere and up
        return 80

    def create_weights(
        self,
        layer: torch.nn.Module,
        num_experts: int,
        hidden_size: int,
        intermediate_size_per_partition: int,
        params_dtype: torch.dtype,
        **extra_weight_attrs,
    ):

View on GitHub (pinned to 0132848349)

Solutions

1. Re-quantize with dynamic per-token input quantization (don't pass static input scales) for per-channel weights
2. Or use per-tensor static input scales instead of channelwise
3. Verify input_quant.dynamic is true and input strategy is TOKEN in the checkpoint

Example fix

// before
"input_quant": {"strategy": "channel", "dynamic": false}
// after
"input_quant": {"strategy": "token", "dynamic": true}

iq = cfg["quantization_config"]["input_quant"]
if not iq.get("dynamic", True):
    assert iq["strategy"] == "tensor", "static input scales require per-tensor for FP8 MoE"

When it happens

Trigger: Passing a checkpoint with static activation scales (from calibration) whose input strategy is CHANNEL/TOKEN to the FP8 MoE scheme: static_input_scales is True and per_channel is True.

Common situations: Calibrated per-channel activation scales exported by older llmcompressor recipes; mixing a static-activation config with a channelwise token-quant recipe.

Related errors

• The hpc_ops MoE runner backend only supports FP8-quantized M
• The output_size of gate's and up's weight = {intermediate_si
• The input_size of down's weight = {intermediate_size_per_par
• Unsupported weight quantization strategy: {self.weight_quant
• QuantConfig has static quantization, but found activation sc

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/ceb139aae09f9197. Report an issue: GitHub.