{"record":{"id":"13edb6084aaebb65","repo":"sgl-project/sglang","slug":"the-output-size-is-not-aligned-with-the-quantized-13edb6","errorCode":null,"errorMessage":"The output size is not aligned with the quantized weight shape. This can be caused by too large tensor parallel size.","messagePattern":"The output size is not aligned with the quantized weight shape\\. This can be caused by too large tensor parallel size\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/gptq/schemes/gptq_cpu.py","lineNumber":72,"sourceCode":"        layer: torch.nn.Module,\n        input_size_per_partition: int,\n        output_partition_sizes: list[int],\n        input_size: int,\n        params_dtype: torch.dtype,\n        weight_loader,\n        **kwargs,\n    ):\n        _check_cpu_amx_support(self.quant_config)\n\n        if input_size_per_partition % self.quant_config.group_size != 0:\n            raise ValueError(\n                \"The input size is not aligned with the quantized \"\n                \"weight shape. This can be caused by too large \"\n                \"tensor parallel size.\"\n            )\n        output_size_per_partition = sum(output_partition_sizes)\n        if output_size_per_partition % self.quant_config.pack_factor.numerator != 0:\n            raise ValueError(\n                \"The output size is not aligned with the quantized \"\n                \"weight shape. This can be caused by too large \"\n                \"tensor parallel size.\"\n            )\n\n        if self.quant_config.group_size != -1:\n            group_size = self.quant_config.group_size\n        else:\n            group_size = input_size\n\n        scale_and_zero_size = input_size_per_partition // group_size\n        scale_and_zero_input_dim = 0\n\n        qweight = PackedvLLMParameter(\n            data=torch.empty(\n                input_size_per_partition // self.quant_config.pack_factor,\n                output_size_per_partition,\n                dtype=torch.int32,","sourceCodeStart":54,"sourceCodeEnd":90,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/gptq/schemes/gptq_cpu.py#L54-L90","documentation":"For GPTQ on CPU/AMX, the per-partition output (N) dimension must be divisible by the pack factor numerator (32/weight_bits), i.e. enough output channels per shard to form complete packed words; TP sharding that leaves a partial pack triggers this check in create_weights.","triggerScenarios":"sum(output_partition_sizes) % quant_config.pack_factor.numerator != 0, e.g. an intermediate size of 11008 with tp=3 yielding a shard not divisible by 8 (for 4-bit pack factor 8), or column-parallel layers sharded across a non-divisor TP degree.","commonSituations":"Same class as the K check: odd tensor-parallel degrees (3, 5, 6, 7) on models whose intermediate/hidden sizes are only divisible by 2/4/8; also fused or stacked projections with unusual output sizes.","solutions":["Change tensor parallel size to a power of two (or any value keeping each shard divisible by the pack factor)","Run with tp=1 on CPU","Verify the model's intermediate_size is divisible by tp * (32/weight_bits)"],"exampleFix":"# before\n--tp 3   # output shard not divisible by pack factor\n\n# after\n--tp 2   # or tp 1","handlingStrategy":"validation","validationCode":"inter, pack_num, tp = model_intermediate_size, 32 // weight_bits, tp_size\nassert inter % (tp * pack_num) == 0, \"output shard not packable; change tp_size\"","typeGuard":"def tp_ok_for_output(out_size: int, pack_numerator: int, tp: int) -> bool:\n    return out_size % tp == 0 and (out_size // tp) % pack_numerator == 0","tryCatchPattern":null,"preventionTips":["Use power-of-two TP degrees for GPTQ models","Check intermediate_size divisibility by tp * (32/weight_bits) before launching"],"tags":["gptq","tensor-parallel","shape-mismatch","pack-factor","cpu","amx"],"backgroundTag":"tensor-parallel-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}