{"record":{"id":"0f2d03a2603f58a6","repo":"sgl-project/sglang","slug":"the-output-size-is-not-aligned-with-the-quantized-0f2d03","errorCode":null,"errorMessage":"The output size is not aligned with the quantized weight shape. This can be caused by too large tensor parallel size.","messagePattern":"The output size is not aligned with the quantized weight shape\\. This can be caused by too large tensor parallel size\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/gptq/schemes/gptq_linear.py","lineNumber":56,"sourceCode":"    def create_weights(\n        self,\n        layer: torch.nn.Module,\n        input_size_per_partition: int,\n        output_partition_sizes: list[int],\n        input_size: int,\n        params_dtype: torch.dtype,\n        weight_loader,\n        **kwargs,\n    ):\n        if input_size_per_partition % self.quant_config.group_size != 0:\n            raise ValueError(\n                \"The input size is not aligned with the quantized \"\n                \"weight shape. This can be caused by too large \"\n                \"tensor parallel size.\"\n            )\n        output_size_per_partition = sum(output_partition_sizes)\n        if output_size_per_partition % self.quant_config.pack_factor.numerator != 0:\n            raise ValueError(\n                \"The output size is not aligned with the quantized \"\n                \"weight shape. This can be caused by too large \"\n                \"tensor parallel size.\"\n            )\n\n        group_size = (\n            self.quant_config.group_size\n            if self.quant_config.group_size != -1\n            else input_size\n        )\n        self.kernel.use_shuffle = True\n        scale_and_zero_size = input_size // group_size\n        scale_and_zero_input_dim = None\n        if (\n            input_size != input_size_per_partition\n            and self.quant_config.group_size != -1\n        ):\n            if self.quant_config.desc_act:","sourceCodeStart":38,"sourceCodeEnd":74,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/gptq/schemes/gptq_linear.py#L38-L74","documentation":"Generic GPTQ linear scheme requires the sharded output dimension to be divisible by the pack factor numerator (32/weight_bits) so packed int32 weights tile correctly across the TP shard.","triggerScenarios":"create_weights where sum(output_partition_sizes) % pack_factor.numerator != 0 — column-parallel or fused projections whose per-rank output channels can't form whole packed groups (e.g. 3-bit pack factor ~10.67 boundaries, or TP=3 on 11008).","commonSituations":"Odd TP degrees, stacked QKV/fused-MoE projections with uneven output partitioning, or nonstandard intermediate sizes in GPTQ checkpoints.","solutions":["Use a tensor parallel size that is a power of two so each output shard is divisible by 32/weight_bits","Set tp=1 for models with unusual projection sizes","Requantize with 4-bit weights so the pack factor divides cleanly"],"exampleFix":"# before\n--tp 7\n\n# after\n--tp 4  # output shard divisible by pack factor 8","handlingStrategy":"validation","validationCode":"pack = 32 // weight_bits\nassert output_size % (tp_size * pack) == 0, \"output shard not divisible by pack factor\"","typeGuard":"def output_packable(out_size: int, pack: int, tp: int) -> bool:\n    return out_size % (tp * pack) == 0","tryCatchPattern":null,"preventionTips":["Avoid odd TP degrees (3,5,6,7) with packed GPTQ weights","Compute pack factor from weight_bits and validate shard sizes at config time"],"tags":["gptq","tensor-parallel","shape-mismatch","pack-factor"],"backgroundTag":"tensor-parallel-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}