sgl-project/sglang · error · Exception

num_bits must be 4 or 8, got {}

Error message

num_bits must be 4 or 8, got {}

What it means

get_weight_perm builds the marlin-style weight permutation for the moe_wna16 GPTQ/AWQ kernels. The interleave pattern differs for 4-bit (8-element interleave) and 8-bit (4-element interleave) weights; any other num_bits value has no defined permutation so the helper raises a bare Exception.

Source

Thrown at python/sglang/srt/layers/quantization/moe_wna16.py:61

        for block in [0, 1]:
            for row in [
                2 * (i % 4),
                2 * (i % 4) + 1,
                2 * (i % 4 + 4),
                2 * (i % 4 + 4) + 1,
            ]:
                perm1.append(16 * row + col + 8 * block)
        for j in range(4):
            perm_list.extend([p + 256 * j for p in perm1])

    perm = np.array(perm_list)

    if num_bits == 4:
        interleave = np.array([0, 2, 4, 6, 1, 3, 5, 7])
    elif num_bits == 8:
        interleave = np.array([0, 2, 1, 3])
    else:
        raise Exception("num_bits must be 4 or 8, got {}".format(num_bits))

    perm = perm.reshape((-1, len(interleave)))[:, interleave].ravel()
    perm = torch.from_numpy(perm)
    return perm


class MoeWNA16Config(QuantizationConfig):
    """Config class for MOE WNA16 (W8A16/W4A16) quantization."""

    def __init__(
        self,
        linear_quant_method: str,
        weight_bits: int,
        group_size: int,
        has_zp: bool,
        lm_head_quantized: bool,
        modules_to_not_convert: Optional[List[str]],
        full_config: Dict[str, Any],

View on GitHub (pinned to 0132848349)

Solutions

1. Re-quantize or use a checkpoint with weight_bits set to 4 or 8 (gptq w4a16 / w8a16 MoE)
2. Fix quant_config.json's bits field if it was edited to an unsupported value
3. If experimenting, add an explicit interleave table entry for your bit-width and verify kernel correctness — do not bypass the check blindly

Example fix

// before
perm = get_weight_perm(num_bits=3, ...)
// after
perm = get_weight_perm(num_bits=4, ...)
// or re-quantize: python -m sglang.srt.quantization ... --bits 4

if num_bits not in (4, 8):
    raise ValueError(f"num_bits {num_bits} unsupported; use 4 or 8")
perm = get_weight_perm(num_bits, perm, sym_temp)

try:
    perm = get_weight_perm(num_bits, ...)
except Exception as e:
    if "num_bits" in str(e):
        raise ValueError("Re-quantize the MoE to 4- or 8-bit weights") from e
    raise

When it happens

Trigger: Calling get_weight_perm with num_bits other than 4 or 8 — e.g. a wna16 config advertising 2-bit, 3-bit, or 16-bit quantization for a MoE layer.

Common situations: Loading a MoE checkpoint quantized to an unusual bit-width (w2a16/w3a16 variants), or hand-editing quant_config.json with bits: 2/3 and expecting the marlin path to handle it.

Related errors

• moe_wna16 only support gptq and awq.
• For Fused MoE layers, only {CompressionFormat.pack_quantized
• num_bits must be 4 or 8, got {}
• Currently, only group size 128 and -1 (channelwise) is suppo
• Weight output_size_per_partition = {output_size_per_partitio

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/1dfc094a382f0a9c. Report an issue: GitHub.