sgl-project/sglang · error · Exception

num_bits must be 4 or 8, got {}

Error message

num_bits must be 4 or 8, got {}

What it means

marlin_zero_points interleaves and packs zero-point tensors for the Marlin GEMM kernel, which is built for exactly 4-bit or 8-bit quantization; the interleaving permutation only exists for those widths. Any other num_bits raises this generic Exception at weight repacking time (called from process_weights_after_loading or the AWQ conversion helpers).

Source

Thrown at python/sglang/srt/layers/quantization/marlin_utils.py:388

        output[e] = marlin_permute_scales(s[e], size_k, size_n, group_size)
    return output


def marlin_zero_points(
    zp: torch.Tensor, size_k: int, size_n: int, num_bits: int
) -> torch.Tensor:
    # Permute zero-points in a similar way to scales, but do not use the
    # "single" permutation, since zero-points are applied on every MMA
    scale_perm, _ = get_scale_perms()
    zp = zp.reshape((-1, len(scale_perm)))[:, scale_perm]

    # Interleave column dim (for the dequantize code) and pack it to int32
    if num_bits == 4:
        interleave = numpy.array([0, 2, 4, 6, 1, 3, 5, 7])
    elif num_bits == 8:
        interleave = numpy.array([0, 2, 1, 3])
    else:
        raise Exception("num_bits must be 4 or 8, got {}".format(num_bits))

    zp = zp.reshape((-1, len(interleave)))[:, interleave].ravel()
    zp = zp.reshape((-1, size_n)).contiguous()
    zp = pack_cols(zp, num_bits, size_k, size_n)

    return zp


def awq_to_marlin_zero_points(
    q_zp_packed: torch.Tensor, size_k: int, size_n: int, num_bits: int
) -> torch.Tensor:
    # AWQ zero-points are quantized and packed on the column dim.
    # In addition, the values are permuted based on dequantizer.
    # Here we undo both of these, and then apply marlin permutation
    # and pack it back.
    q_zp = unpack_cols(q_zp_packed, num_bits, size_k, size_n)

    # Undo interleaving (use argsort(..) to get inverse perm)

View on GitHub (pinned to 0132848349)

Solutions

1. Use a 4-bit or 8-bit quantized checkpoint for the Marlin path
2. Fall back to --quantization gptq / awq (non-Marlin kernels) if the bit width must be kept
3. Check the checkpoint config's bits field resolves to exactly 4 or 8

Example fix

# before
zp = marlin_zero_points(q_zp, k, n, num_bits=3)
# after
zp = marlin_zero_points(q_zp, k, n, num_bits=4)  # use a 4-bit checkpoint

assert num_bits in (4, 8), f"Marlin supports only 4/8 bits, got {num_bits}"

def is_marlin_bits(n) -> bool:
    return n in (4, 8)

When it happens

Trigger: Calling marlin_zero_points / awq_marlin_quantize with num_bits other than 4 or 8 (e.g. 2, 3, 16); feeding a checkpoint whose quant config advertises a non-standard bit width into the Marlin path.

Common situations: Experimenting with 2-bit or 3-bit GPTQ/AWQ checkpoints and expecting the Marlin kernel to handle them; mis-parsing a config where bits is stored as a string or computed value.

Related errors

• Currently, only group size 128 and -1 (channelwise) is suppo
• Weight output_size_per_partition = {output_size_per_partitio
• Weight input_size_per_partition = {input_size_per_partition}
• num_bits must be 4 or 8, got {}
• Unsupported num_bits = {num_bits}. Supported num_bits = {WNA

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/29bbb8eab9124b96. Report an issue: GitHub.