{"record":{"id":"4b2251085bcf955a","repo":"sgl-project/sglang","slug":"weight-input-size-per-partition-input-size-per-4b2251","errorCode":null,"errorMessage":"Weight input_size_per_partition = {input_size_per_partition} is not divisible by group_size = {self.quant_config.group_size}.","messagePattern":"Weight input_size_per_partition = (.+?) is not divisible by group_size = (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/marlin_utils.py","lineNumber":785,"sourceCode":"        if output_size_per_partition % self.quant_config.pack_factor != 0:\n            raise ValueError(\n                f\"Weight output_size_per_partition = \"\n                f\"{output_size_per_partition} is not divisible by \"\n                f\"pack_factor = {self.quant_config.pack_factor}.\"\n            )\n\n        # Validate input_size_per_partition\n        if input_size_per_partition % self.quant_config.min_k_threads != 0:\n            raise ValueError(\n                f\"Weight input_size_per_partition = \"\n                f\"{input_size_per_partition} is not divisible by \"\n                f\"min_k_threads = {self.quant_config.min_k_threads}.\"\n            )\n        if (\n            self.quant_config.group_size != -1\n            and input_size_per_partition % self.quant_config.group_size != 0\n        ):\n            raise ValueError(\n                f\"Weight input_size_per_partition = \"\n                f\"{input_size_per_partition} is not divisible by \"\n                f\"group_size = {self.quant_config.group_size}.\"\n            )\n\n        # Check that we have at least 4 tiles horizontally in the shard\n        num_tiles_per_perm = self.quant_config.perm_len // (\n            self.quant_config.tile_size**2\n        )\n        if output_size_per_partition % num_tiles_per_perm != 0:\n            raise ValueError(\"Each permutation group must reside on the same gpu\")\n\n        # Quantized 4Bit weights packed into Int32.\n        qweight = PackedvLLMParameter(\n            data=torch.empty(\n                input_size_per_partition // self.quant_config.tile_size,\n                output_size_per_partition\n                * self.quant_config.tile_size","sourceCodeStart":767,"sourceCodeEnd":803,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/marlin_utils.py#L767-L803","documentation":"When group_size != -1, Marlin stores one scale per group along K, so input_size_per_partition must be divisible by group_size (typically 128). create_weights raises this if the TP-sharded local K cuts through a quant group, leaving scales unaddressable.","triggerScenarios":"Local K after sharding not divisible by group_size 128, e.g. hidden 5120 with TP=3 giving 1706.67-group misalignment; QKV fused layers whose per-partition K mixes dimensions.","commonSituations":"Non-power-of-two TP; checkpoints quantized with group_size 128 but served with TP degrees that don't divide the K dim into group multiples.","solutions":["Choose TP in {1,2,4,8,...} so local K is a multiple of group_size (128)","Re-quantize channel-wise (group_size -1) if TP flexibility matters more than accuracy","Verify quantization_config.group_size divides hidden_size / TP exactly"],"exampleFix":"# before\n--tensor-parallel-size 3\n# after\n--tensor-parallel-size 4  # local K = hidden/4 stays multiple of 128","handlingStrategy":"validation","validationCode":"gs = cfg.quantization_config.get(\"group_size\", -1)\nif gs != -1:\n    assert (hidden_size // tp) % gs == 0, \"local K cuts a quant group\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep TP a power of two so local K stays a multiple of group_size 128","Or use channel-wise (-1) quantization when TP flexibility is needed"],"tags":["marlin","tensor-parallel","group-size","shape-validation"],"backgroundTag":"tensor-parallel-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}