{"record":{"id":"176a2ddba4b0477a","repo":"sgl-project/sglang","slug":"currently-only-group-size-128-and-1-channelwise","errorCode":null,"errorMessage":"Currently, only group size 128 and -1 (channelwise) is supported for Marlin, but got group_size of {self.group_size}","messagePattern":"Currently, only group size 128 and -1 \\(channelwise\\) is supported for Marlin, but got group_size of (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/marlin_utils.py","lineNumber":643,"sourceCode":"\nclass MarlinConfig(QuantizationConfig):\n    \"\"\"Config class for Marlin.\n\n    Reference: https://github.com/IST-DASLab/marlin/tree/master\n    \"\"\"\n\n    def __init__(\n        self,\n        group_size: int,\n        lm_head_quantized: bool,\n    ) -> None:\n        super().__init__()\n\n        # Group size for the quantization.\n        self.group_size = group_size\n        self.lm_head_quantized = lm_head_quantized\n        if self.group_size != 128 and self.group_size != -1:\n            raise ValueError(\n                \"Currently, only group size 128 and -1 (channelwise) \"\n                \"is supported for Marlin, but got group_size of \"\n                f\"{self.group_size}\"\n            )\n\n        # 4 Bits packed into 32 bit datatype.\n        self.pack_factor = 32 // 4\n\n        # Tile size used by marlin kernels.\n        self.tile_size = 16\n\n        # Min out_features dim\n        self.min_n_threads = 64\n\n        # Min in_features dim\n        self.min_k_threads = 128\n\n        # Max parallel problems to solve at once (improves large","sourceCodeStart":625,"sourceCodeEnd":661,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/marlin_utils.py#L625-L661","documentation":"The Marlin quant config constructor only accepts group_size 128 (128-element quant groups) or -1 (channel-wise/per-column) because the kernel's scale layout is hardcoded for those. Any other group size from the checkpoint (64, 32, 16, 256...) raises this ValueError during config initialization, before any weights load.","triggerScenarios":"Loading a GPTQ/AWQ Marlin-eligible checkpoint quantized with group_size 32, 64, or 256; checkpoints quantized with 'group_size: -1' variants that serialize as 0 instead of -1.","commonSituations":"Re-quantizing models with default GPTQ settings (often group_size 128 but configurable) and picking 64; mixing AutoAWQ outputs (usually 128) with custom calibrations.","solutions":["Re-quantize the model with group_size 128 (or channel-wise, -1)","Download the official GPTQ/AWQ export which uses group_size 128","Check config.json: quantization_config.group_size must be 128 or -1"],"exampleFix":"# quantize (before)\ngptq quantize --group-size 64 ...\n# quantize (after)\ngptq quantize --group-size 128 ...","handlingStrategy":"validation","validationCode":"gs = cfg.quantization_config.get(\"group_size\", -1)\nassert gs in (128, -1), f\"Marlin requires group_size 128 or -1, got {gs}\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always quantize with --group-size 128 for Marlin compatibility","Check quantization_config.group_size before downloading/serving"],"tags":["marlin","gptq","awq","group-size","config-validation"],"backgroundTag":"unsupported-group-size","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}