{"record":{"id":"29bbb8eab9124b96","repo":"sgl-project/sglang","slug":"num-bits-must-be-4-or-8-got","errorCode":null,"errorMessage":"num_bits must be 4 or 8, got {}","messagePattern":"num_bits must be 4 or 8, got (.+?)","errorType":"validation","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/marlin_utils.py","lineNumber":388,"sourceCode":"        output[e] = marlin_permute_scales(s[e], size_k, size_n, group_size)\n    return output\n\n\ndef marlin_zero_points(\n    zp: torch.Tensor, size_k: int, size_n: int, num_bits: int\n) -> torch.Tensor:\n    # Permute zero-points in a similar way to scales, but do not use the\n    # \"single\" permutation, since zero-points are applied on every MMA\n    scale_perm, _ = get_scale_perms()\n    zp = zp.reshape((-1, len(scale_perm)))[:, scale_perm]\n\n    # Interleave column dim (for the dequantize code) and pack it to int32\n    if num_bits == 4:\n        interleave = numpy.array([0, 2, 4, 6, 1, 3, 5, 7])\n    elif num_bits == 8:\n        interleave = numpy.array([0, 2, 1, 3])\n    else:\n        raise Exception(\"num_bits must be 4 or 8, got {}\".format(num_bits))\n\n    zp = zp.reshape((-1, len(interleave)))[:, interleave].ravel()\n    zp = zp.reshape((-1, size_n)).contiguous()\n    zp = pack_cols(zp, num_bits, size_k, size_n)\n\n    return zp\n\n\ndef awq_to_marlin_zero_points(\n    q_zp_packed: torch.Tensor, size_k: int, size_n: int, num_bits: int\n) -> torch.Tensor:\n    # AWQ zero-points are quantized and packed on the column dim.\n    # In addition, the values are permuted based on dequantizer.\n    # Here we undo both of these, and then apply marlin permutation\n    # and pack it back.\n    q_zp = unpack_cols(q_zp_packed, num_bits, size_k, size_n)\n\n    # Undo interleaving (use argsort(..) to get inverse perm)","sourceCodeStart":370,"sourceCodeEnd":406,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/marlin_utils.py#L370-L406","documentation":"marlin_zero_points interleaves and packs zero-point tensors for the Marlin GEMM kernel, which is built for exactly 4-bit or 8-bit quantization; the interleaving permutation only exists for those widths. Any other num_bits raises this generic Exception at weight repacking time (called from process_weights_after_loading or the AWQ conversion helpers).","triggerScenarios":"Calling marlin_zero_points / awq_marlin_quantize with num_bits other than 4 or 8 (e.g. 2, 3, 16); feeding a checkpoint whose quant config advertises a non-standard bit width into the Marlin path.","commonSituations":"Experimenting with 2-bit or 3-bit GPTQ/AWQ checkpoints and expecting the Marlin kernel to handle them; mis-parsing a config where bits is stored as a string or computed value.","solutions":["Use a 4-bit or 8-bit quantized checkpoint for the Marlin path","Fall back to --quantization gptq / awq (non-Marlin kernels) if the bit width must be kept","Check the checkpoint config's bits field resolves to exactly 4 or 8"],"exampleFix":"# before\nzp = marlin_zero_points(q_zp, k, n, num_bits=3)\n# after\nzp = marlin_zero_points(q_zp, k, n, num_bits=4)  # use a 4-bit checkpoint","handlingStrategy":"type-guard","validationCode":"assert num_bits in (4, 8), f\"Marlin supports only 4/8 bits, got {num_bits}\"","typeGuard":"def is_marlin_bits(n) -> bool:\n    return n in (4, 8)","tryCatchPattern":null,"preventionTips":["Validate config quantization bits before weight repacking","Use official 4-bit GPTQ/AWQ exports"],"tags":["marlin","awq","gptq","bit-width","kernel-support"],"backgroundTag":"unsupported-bit-width","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}