{"record":{"id":"6cbc8078f60ed960","repo":"sgl-project/sglang","slug":"the-input-size-is-not-aligned-with-the-quantized-w-6cbc80","errorCode":null,"errorMessage":"The input size is not aligned with the quantized weight shape. This can be caused by too large tensor parallel size.","messagePattern":"The input size is not aligned with the quantized weight shape\\. This can be caused by too large tensor parallel size\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/gptq/schemes/gptq_cpu.py","lineNumber":65,"sourceCode":"    \"\"\"Linear scheme for GPTQ on Intel CPU with AMX.\"\"\"\n\n    def _init_kernel(self, quant_config: GPTQConfig):\n        return GPTQIntelAMXLinearKernel(quant_config)\n\n    def create_weights(\n        self,\n        layer: torch.nn.Module,\n        input_size_per_partition: int,\n        output_partition_sizes: list[int],\n        input_size: int,\n        params_dtype: torch.dtype,\n        weight_loader,\n        **kwargs,\n    ):\n        _check_cpu_amx_support(self.quant_config)\n\n        if input_size_per_partition % self.quant_config.group_size != 0:\n            raise ValueError(\n                \"The input size is not aligned with the quantized \"\n                \"weight shape. This can be caused by too large \"\n                \"tensor parallel size.\"\n            )\n        output_size_per_partition = sum(output_partition_sizes)\n        if output_size_per_partition % self.quant_config.pack_factor.numerator != 0:\n            raise ValueError(\n                \"The output size is not aligned with the quantized \"\n                \"weight shape. This can be caused by too large \"\n                \"tensor parallel size.\"\n            )\n\n        if self.quant_config.group_size != -1:\n            group_size = self.quant_config.group_size\n        else:\n            group_size = input_size\n\n        scale_and_zero_size = input_size_per_partition // group_size","sourceCodeStart":47,"sourceCodeEnd":83,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/gptq/schemes/gptq_cpu.py#L47-L83","documentation":"For GPTQ on CPU/AMX, the per-partition input (K) dimension must be divisible by the quantization group_size; tensor-parallel sharding can make the local input size a non-multiple of group_size, which would break the group-wise quantized weight layout.","triggerScenarios":"create_weights is called with input_size_per_partition = hidden_size / tp_size (or fused-layer input) not divisible by quant_config.group_size (e.g. hidden 11008 with group_size 128 and a TP degree that doesn't divide evenly, or fused MoE/gate layers with odd K).","commonSituations":"Launching with --tp 3/6/7 on models whose hidden dims don't shard into group-size multiples; MoE router or small fused projections with unusual input sizes.","solutions":["Reduce or change tensor parallel size so hidden_size / tp is divisible by group_size (typically a power of 2 <= group_size)","Use tp=1 or a divisor that keeps the local K a multiple of group_size","Use a checkpoint with a compatible group_size (e.g. group_size that divides the sharded K)"],"exampleFix":"# before\npython -m sglang.launch_server --model gptq-model --tp 3   # 11008/3 not divisible by 128\n\n# after\npython -m sglang.launch_server --model gptq-model --tp 2","handlingStrategy":"validation","validationCode":"hidden, group, tp = model_hidden_size, quant_config.group_size, tp_size\nassert hidden % (tp * group) == 0, (\n    f\"input shard {hidden // tp} not divisible by group_size {group}; pick another tp\")","typeGuard":"def tp_ok_for_input(hidden: int, group: int, tp: int) -> bool:\n    return hidden % tp == 0 and (hidden // tp) % group == 0","tryCatchPattern":null,"preventionTips":["Prefer power-of-two TP sizes with group_size 128","Add a startup assert hidden % (tp * group_size) == 0 in deployment scripts"],"tags":["gptq","tensor-parallel","shape-mismatch","quantization","cpu","amx"],"backgroundTag":"tensor-parallel-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}