sgl-project/sglang · error · ValueError

For Fused MoE layers, only {CompressionFormat.pack_quantized

Error message

For Fused MoE layers, only {CompressionFormat.pack_quantized.value} is supported for the mxint4

What it means

Raised when constructing the W4A4 mxint4 MoE quantization scheme if the checkpoint's compressed-tensors quant_format is not 'pack-quantized'. The mxint4 MoE kernels (flashinfer trtllm backend) only consume weights stored in the packed-quantized format, so any other CompressionFormat is rejected at scheme init.

Source

Thrown at python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a4_mxint4_moe.py:78

        self.strategy = config.strategy
        self.group_size = config.group_size
        self.actorder = config.actorder
        assert (
            config.strategy == "group"
            and config.group_size == 32
            and config.num_bits == 4
        ), "MxInt4 only supports group strategy with group size 32"
        assert config.symmetric, "Only symmetric quantization is supported for MoE"
        assert (
            get_moe_runner_backend().is_flashinfer_trtllm()
        ), "MxInt4 only supports flashinfer_trtllm backend"
        assert (
            not config.actorder
        ), "Actorder is not supported by flashinfer_trtllm backend"
        self.moe_ep_rank = get_parallel().moe_ep_rank

        if self.quant_config.quant_format != CompressionFormat.pack_quantized.value:
            raise ValueError(
                f"For Fused MoE layers, only {CompressionFormat.pack_quantized.value} "
                "is supported for the mxint4"
            )
        self._cache_permute_indices = {}

    @classmethod
    def get_min_capability(cls) -> int:
        # Requires sm100(blackwell) architecture
        return 100

    def create_weights(
        self,
        layer: torch.nn.Module,
        num_experts: int,
        hidden_size: int,
        intermediate_size_per_partition: int,
        params_dtype: torch.dtype,
        **extra_weight_attrs,

View on GitHub (pinned to 0132848349)

Solutions

1. Re-quantize the model with llmcompressor exporting CompressionFormat.pack_quantized for mxint4 weights
2. Check config.json: quant_config.quant_format should read "pack-quantized"
3. If the model is not truly mxint4, ensure the scheme selector isn't misrouting it (verify weight bit-width/dtype)

Example fix

// before: config.json has "format": "int-quantized"
// after:
"quantization_config": {"format": "pack-quantized", "weights": {"num_bits": 8, "type": "mxint4"}}

from compressed_tensors import CompressionFormat
q = model_config.quantization_config
assert q["format"] == CompressionFormat.pack_quantized.value, f"mxint4 MoE needs pack-quantized, got {q['format']}"

When it happens

Trigger: Loading a model whose config.json compressed-tensors section uses a format like 'dense' or 'int-quantized' while the quantization code selects CompressedTensorsW4A4Mxint4MoE (mxint4 weights + Fused MoE layer).

Common situations: Quantizing a MoE model with llmcompressor to mxint4 but exporting a non-packed format; hand-editing quant config; using a checkpoint produced by an older/newer tool version with different format names.

Related errors

• The Triton WNA16 MoE backend only supports symmetric INT4 gr
• Unsupported FusedMoe scheme: {weight_quant}, {input_quant}
• Unsupported weight quantization strategy: {self.weight_quant
• For Fused MoE layers, only {CompressionFormat.pack_quantized
• Config list contains configs from 2 methods, must be only 1

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7f2370dc6108f733. Report an issue: GitHub.