{"record":{"id":"982ef3069d7c73c2","repo":"sgl-project/sglang","slug":"for-int8-fused-moe-layers-we-require-channelwise","errorCode":null,"errorMessage":"For INT8 Fused MoE layers, we require channelwise, dynamic per token quantization. Found {self.weight_quant}, {self.input_quant}","messagePattern":"For INT8 Fused MoE layers, we require channelwise, dynamic per token quantization\\. Found (.+?), (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8_moe.py","lineNumber":44,"sourceCode":"\nlogger = logging.getLogger(__name__)\n\n\nclass NPUCompressedTensorsW8A8Int8DynamicMoE(CompressedTensorsMoEScheme):\n\n    def __init__(self, weight_quant, input_quant):\n        self.weight_quant = weight_quant\n        self.input_quant = input_quant\n        self.w13_kernel = NPUW8A8Int8MoEMethod()\n        self.w2_kernel = NPUW8A8Int8MoEMethod()\n\n        self.static_input_scales = not self.input_quant.dynamic\n        per_channel = (\n            self.weight_quant.strategy == QuantizationStrategy.CHANNEL\n            and self.input_quant.strategy == QuantizationStrategy.TOKEN\n        )\n        if not per_channel:\n            raise ValueError(\n                \"For INT8 Fused MoE layers, we require channelwise, \"\n                \"dynamic per token quantization. Found \"\n                f\"{self.weight_quant}, {self.input_quant}\"\n            )\n\n        self.static_input_scales = not self.input_quant.dynamic\n        if self.static_input_scales:\n            raise ValueError(\n                \"For INT8 Fused MoE layers, we require channelwise, \"\n                \"dynamic per token quantization. Found static input scales.\"\n            )\n\n    def create_weights(\n        self,\n        layer: torch.nn.Module,\n        num_experts: int,\n        hidden_size: int,\n        intermediate_size_per_partition: int,","sourceCodeStart":26,"sourceCodeEnd":62,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8_moe.py#L26-L62","documentation":"INT8 Fused MoE requires exactly channelwise weight quantization combined with dynamic per-token input quantization. This error fires when weight strategy != CHANNEL or input strategy != TOKEN, printing both configs to show the mismatch.","triggerScenarios":"Loading a compressed-tensors INT8 MoE model whose weights are per-tensor or grouped, or whose activations are per-tensor, in CompressedTensorsW8A8Int8MoE.__init__.","commonSituations":"Applying a generic INT8 recipe (e.g. per-tensor weights) to a MoE model; recipes tuned for linear layers reused on MoE.","solutions":["Re-quantize with weights strategy=channel and input strategy=token, dynamic=true","Use the FP8 MoE path if the recipe can't be changed","Check the printed weight_quant/input_quant in the error output against the requirement"],"exampleFix":"// before\n\"weights\": {\"strategy\": \"tensor\"}\n// after\n\"weights\": {\"strategy\": \"channel\"}, \"input_quant\": {\"strategy\": \"token\", \"dynamic\": true}","handlingStrategy":"validation","validationCode":"w, iq = cfg[\"weights\"], cfg[\"input_quant\"]\nassert w[\"strategy\"] == \"channel\" and iq[\"strategy\"] == \"token\", \"INT8 MoE needs channel weights + token activations\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use the canonical INT8 MoE recipe (channel weights, dynamic token inputs)"],"tags":["quantization","int8","moe","strategy"],"backgroundTag":"invalid-quantization-config-combo","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}