sgl-project/sglang · error · ValueError

The input size is not aligned with the quantized weight shap

Error message

The input size is not aligned with the quantized weight shape. This can be caused by too large tensor parallel size.

What it means

For GPTQ on CPU/AMX, the per-partition input (K) dimension must be divisible by the quantization group_size; tensor-parallel sharding can make the local input size a non-multiple of group_size, which would break the group-wise quantized weight layout.

Source

Thrown at python/sglang/srt/layers/quantization/gptq/schemes/gptq_cpu.py:65

    """Linear scheme for GPTQ on Intel CPU with AMX."""

    def _init_kernel(self, quant_config: GPTQConfig):
        return GPTQIntelAMXLinearKernel(quant_config)

    def create_weights(
        self,
        layer: torch.nn.Module,
        input_size_per_partition: int,
        output_partition_sizes: list[int],
        input_size: int,
        params_dtype: torch.dtype,
        weight_loader,
        **kwargs,
    ):
        _check_cpu_amx_support(self.quant_config)

        if input_size_per_partition % self.quant_config.group_size != 0:
            raise ValueError(
                "The input size is not aligned with the quantized "
                "weight shape. This can be caused by too large "
                "tensor parallel size."
            )
        output_size_per_partition = sum(output_partition_sizes)
        if output_size_per_partition % self.quant_config.pack_factor.numerator != 0:
            raise ValueError(
                "The output size is not aligned with the quantized "
                "weight shape. This can be caused by too large "
                "tensor parallel size."
            )

        if self.quant_config.group_size != -1:
            group_size = self.quant_config.group_size
        else:
            group_size = input_size

        scale_and_zero_size = input_size_per_partition // group_size

View on GitHub (pinned to 0132848349)

Solutions

1. Reduce or change tensor parallel size so hidden_size / tp is divisible by group_size (typically a power of 2 <= group_size)
2. Use tp=1 or a divisor that keeps the local K a multiple of group_size
3. Use a checkpoint with a compatible group_size (e.g. group_size that divides the sharded K)

Example fix

# before
python -m sglang.launch_server --model gptq-model --tp 3   # 11008/3 not divisible by 128

# after
python -m sglang.launch_server --model gptq-model --tp 2

hidden, group, tp = model_hidden_size, quant_config.group_size, tp_size
assert hidden % (tp * group) == 0, (
    f"input shard {hidden // tp} not divisible by group_size {group}; pick another tp")

def tp_ok_for_input(hidden: int, group: int, tp: int) -> bool:
    return hidden % tp == 0 and (hidden // tp) % group == 0

When it happens

Trigger: create_weights is called with input_size_per_partition = hidden_size / tp_size (or fused-layer input) not divisible by quant_config.group_size (e.g. hidden 11008 with group_size 128 and a TP degree that doesn't divide evenly, or fused MoE/gate layers with odd K).

Common situations: Launching with --tp 3/6/7 on models whose hidden dims don't shard into group-size multiples; MoE router or small fused projections with unusual input sizes.

Related errors

• The output size is not aligned with the quantized weight sha
• Currently, only 4bits is supported on CPU with AMX.
• Weight output_partition_size = {output_partition_size} is no
• GPTQ act_order on XPU requires each group_size block of inpu
• Invalid quantization method on CPU: {quantization}. Availabl

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6cbc8078f60ed960. Report an issue: GitHub.