sgl-project/sglang · error · ValueError

Currently, only group size 128 and -1 (channelwise) is suppo

Error message

Currently, only group size 128 and -1 (channelwise) is supported for Marlin, but got group_size of {self.group_size}

What it means

The Marlin quant config constructor only accepts group_size 128 (128-element quant groups) or -1 (channel-wise/per-column) because the kernel's scale layout is hardcoded for those. Any other group size from the checkpoint (64, 32, 16, 256...) raises this ValueError during config initialization, before any weights load.

Source

Thrown at python/sglang/srt/layers/quantization/marlin_utils.py:643

class MarlinConfig(QuantizationConfig):
    """Config class for Marlin.

    Reference: https://github.com/IST-DASLab/marlin/tree/master
    """

    def __init__(
        self,
        group_size: int,
        lm_head_quantized: bool,
    ) -> None:
        super().__init__()

        # Group size for the quantization.
        self.group_size = group_size
        self.lm_head_quantized = lm_head_quantized
        if self.group_size != 128 and self.group_size != -1:
            raise ValueError(
                "Currently, only group size 128 and -1 (channelwise) "
                "is supported for Marlin, but got group_size of "
                f"{self.group_size}"
            )

        # 4 Bits packed into 32 bit datatype.
        self.pack_factor = 32 // 4

        # Tile size used by marlin kernels.
        self.tile_size = 16

        # Min out_features dim
        self.min_n_threads = 64

        # Min in_features dim
        self.min_k_threads = 128

        # Max parallel problems to solve at once (improves large

View on GitHub (pinned to 0132848349)

Solutions

1. Re-quantize the model with group_size 128 (or channel-wise, -1)
2. Download the official GPTQ/AWQ export which uses group_size 128
3. Check config.json: quantization_config.group_size must be 128 or -1

Example fix

# quantize (before)
gptq quantize --group-size 64 ...
# quantize (after)
gptq quantize --group-size 128 ...

gs = cfg.quantization_config.get("group_size", -1)
assert gs in (128, -1), f"Marlin requires group_size 128 or -1, got {gs}"

When it happens

Trigger: Loading a GPTQ/AWQ Marlin-eligible checkpoint quantized with group_size 32, 64, or 256; checkpoints quantized with 'group_size: -1' variants that serialize as 0 instead of -1.

Common situations: Re-quantizing models with default GPTQ settings (often group_size 128 but configurable) and picking 64; mixing AutoAWQ outputs (usually 128) with custom calibrations.

Related errors

• num_bits must be 4 or 8, got {}
• Weight output_size_per_partition = {output_size_per_partitio
• Weight input_size_per_partition = {input_size_per_partition}
• num_bits must be 4 or 8, got {}
• moe_wna16 only support gptq and awq.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/176a2ddba4b0477a. Report an issue: GitHub.