{"record":{"id":"7f2370dc6108f733","repo":"sgl-project/sglang","slug":"for-fused-moe-layers-only-compressionformat-pack","errorCode":null,"errorMessage":"For Fused MoE layers, only {CompressionFormat.pack_quantized.value} is supported for the mxint4","messagePattern":"For Fused MoE layers, only (.+?) is supported for the mxint4","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a4_mxint4_moe.py","lineNumber":78,"sourceCode":"        self.strategy = config.strategy\n        self.group_size = config.group_size\n        self.actorder = config.actorder\n        assert (\n            config.strategy == \"group\"\n            and config.group_size == 32\n            and config.num_bits == 4\n        ), \"MxInt4 only supports group strategy with group size 32\"\n        assert config.symmetric, \"Only symmetric quantization is supported for MoE\"\n        assert (\n            get_moe_runner_backend().is_flashinfer_trtllm()\n        ), \"MxInt4 only supports flashinfer_trtllm backend\"\n        assert (\n            not config.actorder\n        ), \"Actorder is not supported by flashinfer_trtllm backend\"\n        self.moe_ep_rank = get_parallel().moe_ep_rank\n\n        if self.quant_config.quant_format != CompressionFormat.pack_quantized.value:\n            raise ValueError(\n                f\"For Fused MoE layers, only {CompressionFormat.pack_quantized.value} \"\n                \"is supported for the mxint4\"\n            )\n        self._cache_permute_indices = {}\n\n    @classmethod\n    def get_min_capability(cls) -> int:\n        # Requires sm100(blackwell) architecture\n        return 100\n\n    def create_weights(\n        self,\n        layer: torch.nn.Module,\n        num_experts: int,\n        hidden_size: int,\n        intermediate_size_per_partition: int,\n        params_dtype: torch.dtype,\n        **extra_weight_attrs,","sourceCodeStart":60,"sourceCodeEnd":96,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a4_mxint4_moe.py#L60-L96","documentation":"Raised when constructing the W4A4 mxint4 MoE quantization scheme if the checkpoint's compressed-tensors quant_format is not 'pack-quantized'. The mxint4 MoE kernels (flashinfer trtllm backend) only consume weights stored in the packed-quantized format, so any other CompressionFormat is rejected at scheme init.","triggerScenarios":"Loading a model whose config.json compressed-tensors section uses a format like 'dense' or 'int-quantized' while the quantization code selects CompressedTensorsW4A4Mxint4MoE (mxint4 weights + Fused MoE layer).","commonSituations":"Quantizing a MoE model with llmcompressor to mxint4 but exporting a non-packed format; hand-editing quant config; using a checkpoint produced by an older/newer tool version with different format names.","solutions":["Re-quantize the model with llmcompressor exporting CompressionFormat.pack_quantized for mxint4 weights","Check config.json: quant_config.quant_format should read \"pack-quantized\"","If the model is not truly mxint4, ensure the scheme selector isn't misrouting it (verify weight bit-width/dtype)"],"exampleFix":"// before: config.json has \"format\": \"int-quantized\"\n// after:\n\"quantization_config\": {\"format\": \"pack-quantized\", \"weights\": {\"num_bits\": 8, \"type\": \"mxint4\"}}","handlingStrategy":"validation","validationCode":"from compressed_tensors import CompressionFormat\nq = model_config.quantization_config\nassert q[\"format\"] == CompressionFormat.pack_quantized.value, f\"mxint4 MoE needs pack-quantized, got {q['format']}\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Validate quantization_config.format before launching the server","Use llmcompressor export presets known to produce pack-quantized mxint4"],"tags":["quantization","moe","mxint4","compressed-tensors","model-config"],"backgroundTag":"unsupported-quantization-format","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}