{"record":{"id":"9d41160134864526","repo":"sgl-project/sglang","slug":"marlin-kernels-require-group-quantization-or-chann","errorCode":null,"errorMessage":"Marlin kernels require group quantization or channelwise quantization, but found no group size and strategy is not channelwise.","messagePattern":"Marlin kernels require group quantization or channelwise quantization, but found no group size and strategy is not channelwise\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_wNa16.py","lineNumber":80,"sourceCode":"\nclass CompressedTensorsWNA16(CompressedTensorsLinearScheme):\n    _kernel_backends_being_used: set[str] = set()\n\n    def __init__(self,\n                 strategy: str,\n                 num_bits: int,\n                 group_size: Optional[int] = None,\n                 symmetric: Optional[bool] = True,\n                 actorder: Optional[ActivationOrdering] = None):\n\n        self.pack_factor = 32 // num_bits\n        self.strategy = strategy\n        self.symmetric = symmetric\n        self.group_size = -1 if group_size is None else group_size\n        self.has_g_idx = actorder == ActivationOrdering.GROUP\n\n        if self.group_size == -1 and self.strategy != \"channel\":\n            raise ValueError(\"Marlin kernels require group quantization or \"\n                             \"channelwise quantization, but found no group \"\n                             \"size and strategy is not channelwise.\")\n\n        if num_bits not in WNA16_SUPPORTED_TYPES_MAP:\n            raise ValueError(\n                f\"Unsupported num_bits = {num_bits}. \"\n                f\"Supported num_bits = {WNA16_SUPPORTED_TYPES_MAP.keys()}\")\n\n        self.quant_type = (WNA16_ZP_SUPPORTED_TYPES_MAP[num_bits]\n                           if not self.symmetric else\n                           WNA16_SUPPORTED_TYPES_MAP[num_bits])\n\n    @classmethod\n    def get_min_capability(cls) -> int:\n        # ampere and up\n        return 80\n\n    def create_weights(self, layer: torch.nn.Module, output_size: int,","sourceCodeStart":62,"sourceCodeEnd":98,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_wNa16.py#L62-L98","documentation":"Marlin wNa16 kernels need either group-wise quantization (a group_size) or channelwise strategy. A checkpoint with no group_size and a non-channel strategy (e.g. per-tensor) cannot be repacked for Marlin, so scheme init fails immediately.","triggerScenarios":"Loading a GPTQ/AWQ-style compressed-tensors model whose weights section has strategy 'tensor' and no group_size, routed to CompressedTensorsWNA16.","commonSituations":"Per-tensor INT4/INT8-quantized checkpoints mistakenly run through the Marlin path; quant configs missing the group_size field.","solutions":["Re-quantize with group_size set (e.g. 32/64/128) or channelwise strategy","Verify config weights: {\"strategy\": \"group\", \"group_size\": 128}","If per-tensor is required, use a non-Marlin-compatible scheme or convert the checkpoint"],"exampleFix":"// before\n\"weights\": {\"num_bits\": 4, \"strategy\": \"tensor\"}\n// after\n\"weights\": {\"num_bits\": 4, \"strategy\": \"group\", \"group_size\": 128}","handlingStrategy":"validation","validationCode":"w = cfg[\"quantization_config\"][\"weights\"]\nassert w.get(\"group_size\") is not None or w[\"strategy\"] == \"channel\"","typeGuard":"def marlin_compatible(w):\n    return w.get(\"group_size\") is not None or w[\"strategy\"] == \"channel\"","tryCatchPattern":null,"preventionTips":["Always set group_size when quantizing for Marlin kernels"],"tags":["quantization","marlin","int4","group-quantization","strategy"],"backgroundTag":"invalid-quantization-config-combo","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}