sgl-project/sglang · error · ValueError

The input size is not aligned with the quantized weight shap

Error message

The input size is not aligned with the quantized weight shape. This can be caused by too large tensor parallel size.

What it means

Generic (non-CPU) GPTQ linear scheme requires input_size_per_partition (local K = input_size / tp) to be divisible by quant_config.group_size; otherwise the group-wise scales/qzeros layout cannot be constructed.

Source

Thrown at python/sglang/srt/layers/quantization/gptq/schemes/gptq_linear.py:49

    def _init_kernel(self, quant_config: GPTQConfig):
        from sglang.srt.hardware_backend.gpu.quantization.gptq_kernels import (
            GPTQLinearKernel,
        )

        return GPTQLinearKernel(quant_config)

    def create_weights(
        self,
        layer: torch.nn.Module,
        input_size_per_partition: int,
        output_partition_sizes: list[int],
        input_size: int,
        params_dtype: torch.dtype,
        weight_loader,
        **kwargs,
    ):
        if input_size_per_partition % self.quant_config.group_size != 0:
            raise ValueError(
                "The input size is not aligned with the quantized "
                "weight shape. This can be caused by too large "
                "tensor parallel size."
            )
        output_size_per_partition = sum(output_partition_sizes)
        if output_size_per_partition % self.quant_config.pack_factor.numerator != 0:
            raise ValueError(
                "The output size is not aligned with the quantized "
                "weight shape. This can be caused by too large "
                "tensor parallel size."
            )

        group_size = (
            self.quant_config.group_size
            if self.quant_config.group_size != -1
            else input_size
        )
        self.kernel.use_shuffle = True

View on GitHub (pinned to 0132848349)

Solutions

1. Pick a tensor parallel size such that hidden_size % (tp * group_size) == 0 (usually tp in {1,2,4,8})
2. Set group_size in the quantization config to -1 (per-channel) or a divisor of the local K if requantizing
3. Requantize the model with a group_size compatible with the intended TP layout

Example fix

# before
server_args = ServerArgs(model_path="gptq-model", tp_size=6)

# after
server_args = ServerArgs(model_path="gptq-model", tp_size=4)

assert input_size % (tp_size * group_size) == 0, (
    f"{input_size} / tp{tp_size} breaks group_size {group_size} alignment")

def shards_preserve_groups(in_size: int, group: int, tp: int) -> bool:
    return in_size % (tp * group) == 0

When it happens

Trigger: create_weights with a layer whose sharded input dimension isn't a multiple of group_size — typical for row-parallel layers under TP degrees that don't divide hidden_size into group-size multiples, or group_size=-1 handled elsewhere with a bad fallback.

Common situations: Launching GPTQ models with --tp values like 3/6/7; models with non-standard hidden sizes (e.g. some MoE experts) where TP shards break quantization groups.

Related errors

• The input size is not aligned with the quantized weight shap
• The output size is not aligned with the quantized weight sha
• The output size is not aligned with the quantized weight sha
• W4AFP8 shape_k = {shape_k} must be divisible by group_size =
• Weight output_partition_size = {output_partition_size} is no

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b1a76a3ddedc7001. Report an issue: GitHub.