{"record":{"id":"ceb139aae09f9197","repo":"sgl-project/sglang","slug":"for-fp8-fused-moe-layer-we-require-either-per-ten","errorCode":null,"errorMessage":"For FP8 Fused MoE layer, we require either per tensor or channelwise, dynamic per token quantization.","messagePattern":"For FP8 Fused MoE layer, we require either per tensor or channelwise, dynamic per token quantization\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8_moe.py","lineNumber":76,"sourceCode":"        per_tensor = (\n            self.weight_quant.strategy == QuantizationStrategy.TENSOR\n            and self.input_quant.strategy == QuantizationStrategy.TENSOR\n        )\n        per_channel = (\n            self.weight_quant.strategy == QuantizationStrategy.CHANNEL\n            and self.input_quant.strategy == QuantizationStrategy.TOKEN\n        )\n        if not (per_tensor or per_channel):\n            assert self.weight_quant.strategy == QuantizationStrategy.BLOCK\n            self.weight_block_size = self.weight_quant.block_structure\n            assert self.weight_quant.dynamic is not None\n        else:\n            self.weight_block_size = None\n        self.block_quant = self.weight_block_size is not None\n\n        self.static_input_scales = not self.input_quant.dynamic\n        if self.static_input_scales and per_channel:\n            raise ValueError(\n                \"For FP8 Fused MoE layer, we require either per tensor or \"\n                \"channelwise, dynamic per token quantization.\"\n            )\n\n    @classmethod\n    def get_min_capability(cls) -> int:\n        # ampere and up\n        return 80\n\n    def create_weights(\n        self,\n        layer: torch.nn.Module,\n        num_experts: int,\n        hidden_size: int,\n        intermediate_size_per_partition: int,\n        params_dtype: torch.dtype,\n        **extra_weight_attrs,\n    ):","sourceCodeStart":58,"sourceCodeEnd":94,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8_moe.py#L58-L94","documentation":"For FP8 Fused MoE, static (calibrated) input scales are only supported with per-tensor input quantization. This error fires when input scales are static (input_quant.dynamic == False) AND the quantization is per-channel/per-token — a combination the fused MoE kernels cannot consume.","triggerScenarios":"Passing a checkpoint with static activation scales (from calibration) whose input strategy is CHANNEL/TOKEN to the FP8 MoE scheme: static_input_scales is True and per_channel is True.","commonSituations":"Calibrated per-channel activation scales exported by older llmcompressor recipes; mixing a static-activation config with a channelwise token-quant recipe.","solutions":["Re-quantize with dynamic per-token input quantization (don't pass static input scales) for per-channel weights","Or use per-tensor static input scales instead of channelwise","Verify input_quant.dynamic is true and input strategy is TOKEN in the checkpoint"],"exampleFix":"// before\n\"input_quant\": {\"strategy\": \"channel\", \"dynamic\": false}\n// after\n\"input_quant\": {\"strategy\": \"token\", \"dynamic\": true}","handlingStrategy":"validation","validationCode":"iq = cfg[\"quantization_config\"][\"input_quant\"]\nif not iq.get(\"dynamic\", True):\n    assert iq[\"strategy\"] == \"tensor\", \"static input scales require per-tensor for FP8 MoE\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Prefer dynamic per-token input quantization for MoE models","Review calibration recipes before exporting static scales"],"tags":["quantization","fp8","moe","static-scales","input-quantization"],"backgroundTag":"invalid-quantization-config-combo","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}