sgl-project/sglang · error · ValueError

The output size is not aligned with the quantized weight sha

Error message

The output size is not aligned with the quantized weight shape. This can be caused by too large tensor parallel size.

What it means

Generic GPTQ linear scheme requires the sharded output dimension to be divisible by the pack factor numerator (32/weight_bits) so packed int32 weights tile correctly across the TP shard.

Source

Thrown at python/sglang/srt/layers/quantization/gptq/schemes/gptq_linear.py:56

    def create_weights(
        self,
        layer: torch.nn.Module,
        input_size_per_partition: int,
        output_partition_sizes: list[int],
        input_size: int,
        params_dtype: torch.dtype,
        weight_loader,
        **kwargs,
    ):
        if input_size_per_partition % self.quant_config.group_size != 0:
            raise ValueError(
                "The input size is not aligned with the quantized "
                "weight shape. This can be caused by too large "
                "tensor parallel size."
            )
        output_size_per_partition = sum(output_partition_sizes)
        if output_size_per_partition % self.quant_config.pack_factor.numerator != 0:
            raise ValueError(
                "The output size is not aligned with the quantized "
                "weight shape. This can be caused by too large "
                "tensor parallel size."
            )

        group_size = (
            self.quant_config.group_size
            if self.quant_config.group_size != -1
            else input_size
        )
        self.kernel.use_shuffle = True
        scale_and_zero_size = input_size // group_size
        scale_and_zero_input_dim = None
        if (
            input_size != input_size_per_partition
            and self.quant_config.group_size != -1
        ):
            if self.quant_config.desc_act:

View on GitHub (pinned to 0132848349)

Solutions

1. Use a tensor parallel size that is a power of two so each output shard is divisible by 32/weight_bits
2. Set tp=1 for models with unusual projection sizes
3. Requantize with 4-bit weights so the pack factor divides cleanly

Example fix

# before
--tp 7

# after
--tp 4  # output shard divisible by pack factor 8

pack = 32 // weight_bits
assert output_size % (tp_size * pack) == 0, "output shard not divisible by pack factor"

def output_packable(out_size: int, pack: int, tp: int) -> bool:
    return out_size % (tp * pack) == 0

When it happens

Trigger: create_weights where sum(output_partition_sizes) % pack_factor.numerator != 0 — column-parallel or fused projections whose per-rank output channels can't form whole packed groups (e.g. 3-bit pack factor ~10.67 boundaries, or TP=3 on 11008).

Common situations: Odd TP degrees, stacked QKV/fused-MoE projections with uneven output partitioning, or nonstandard intermediate sizes in GPTQ checkpoints.

Related errors

• The output size is not aligned with the quantized weight sha
• The input size is not aligned with the quantized weight shap
• The input size is not aligned with the quantized weight shap
• Weight output_partition_size = {output_partition_size} is no
• GPTQ act_order on XPU requires each group_size block of inpu

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0f2d03a2603f58a6. Report an issue: GitHub.