{"record":{"id":"b1a76a3ddedc7001","repo":"sgl-project/sglang","slug":"the-input-size-is-not-aligned-with-the-quantized-w-b1a76a","errorCode":null,"errorMessage":"The input size is not aligned with the quantized weight shape. This can be caused by too large tensor parallel size.","messagePattern":"The input size is not aligned with the quantized weight shape\\. This can be caused by too large tensor parallel size\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/gptq/schemes/gptq_linear.py","lineNumber":49,"sourceCode":"    def _init_kernel(self, quant_config: GPTQConfig):\n        from sglang.srt.hardware_backend.gpu.quantization.gptq_kernels import (\n            GPTQLinearKernel,\n        )\n\n        return GPTQLinearKernel(quant_config)\n\n    def create_weights(\n        self,\n        layer: torch.nn.Module,\n        input_size_per_partition: int,\n        output_partition_sizes: list[int],\n        input_size: int,\n        params_dtype: torch.dtype,\n        weight_loader,\n        **kwargs,\n    ):\n        if input_size_per_partition % self.quant_config.group_size != 0:\n            raise ValueError(\n                \"The input size is not aligned with the quantized \"\n                \"weight shape. This can be caused by too large \"\n                \"tensor parallel size.\"\n            )\n        output_size_per_partition = sum(output_partition_sizes)\n        if output_size_per_partition % self.quant_config.pack_factor.numerator != 0:\n            raise ValueError(\n                \"The output size is not aligned with the quantized \"\n                \"weight shape. This can be caused by too large \"\n                \"tensor parallel size.\"\n            )\n\n        group_size = (\n            self.quant_config.group_size\n            if self.quant_config.group_size != -1\n            else input_size\n        )\n        self.kernel.use_shuffle = True","sourceCodeStart":31,"sourceCodeEnd":67,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/gptq/schemes/gptq_linear.py#L31-L67","documentation":"Generic (non-CPU) GPTQ linear scheme requires input_size_per_partition (local K = input_size / tp) to be divisible by quant_config.group_size; otherwise the group-wise scales/qzeros layout cannot be constructed.","triggerScenarios":"create_weights with a layer whose sharded input dimension isn't a multiple of group_size — typical for row-parallel layers under TP degrees that don't divide hidden_size into group-size multiples, or group_size=-1 handled elsewhere with a bad fallback.","commonSituations":"Launching GPTQ models with --tp values like 3/6/7; models with non-standard hidden sizes (e.g. some MoE experts) where TP shards break quantization groups.","solutions":["Pick a tensor parallel size such that hidden_size % (tp * group_size) == 0 (usually tp in {1,2,4,8})","Set group_size in the quantization config to -1 (per-channel) or a divisor of the local K if requantizing","Requantize the model with a group_size compatible with the intended TP layout"],"exampleFix":"# before\nserver_args = ServerArgs(model_path=\"gptq-model\", tp_size=6)\n\n# after\nserver_args = ServerArgs(model_path=\"gptq-model\", tp_size=4)","handlingStrategy":"validation","validationCode":"assert input_size % (tp_size * group_size) == 0, (\n    f\"{input_size} / tp{tp_size} breaks group_size {group_size} alignment\")","typeGuard":"def shards_preserve_groups(in_size: int, group: int, tp: int) -> bool:\n    return in_size % (tp * group) == 0","tryCatchPattern":null,"preventionTips":["Restrict GPTQ deployments to tp in {1,2,4,8} unless you've verified divisibility","Write a preflight script that reads model dims + quant config and validates TP choices"],"tags":["gptq","tensor-parallel","shape-mismatch","group-size"],"backgroundTag":"tensor-parallel-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}