sgl-project/sglang · error · ValueError

The output size is not aligned with the quantized weight sha

Error message

The output size is not aligned with the quantized weight shape. This can be caused by too large tensor parallel size.

What it means

For GPTQ on CPU/AMX, the per-partition output (N) dimension must be divisible by the pack factor numerator (32/weight_bits), i.e. enough output channels per shard to form complete packed words; TP sharding that leaves a partial pack triggers this check in create_weights.

Source

Thrown at python/sglang/srt/layers/quantization/gptq/schemes/gptq_cpu.py:72

        layer: torch.nn.Module,
        input_size_per_partition: int,
        output_partition_sizes: list[int],
        input_size: int,
        params_dtype: torch.dtype,
        weight_loader,
        **kwargs,
    ):
        _check_cpu_amx_support(self.quant_config)

        if input_size_per_partition % self.quant_config.group_size != 0:
            raise ValueError(
                "The input size is not aligned with the quantized "
                "weight shape. This can be caused by too large "
                "tensor parallel size."
            )
        output_size_per_partition = sum(output_partition_sizes)
        if output_size_per_partition % self.quant_config.pack_factor.numerator != 0:
            raise ValueError(
                "The output size is not aligned with the quantized "
                "weight shape. This can be caused by too large "
                "tensor parallel size."
            )

        if self.quant_config.group_size != -1:
            group_size = self.quant_config.group_size
        else:
            group_size = input_size

        scale_and_zero_size = input_size_per_partition // group_size
        scale_and_zero_input_dim = 0

        qweight = PackedvLLMParameter(
            data=torch.empty(
                input_size_per_partition // self.quant_config.pack_factor,
                output_size_per_partition,
                dtype=torch.int32,

View on GitHub (pinned to 0132848349)

Solutions

Change tensor parallel size to a power of two (or any value keeping each shard divisible by the pack factor)
Run with tp=1 on CPU
Verify the model's intermediate_size is divisible by tp * (32/weight_bits)

Example fix

# before
--tp 3   # output shard not divisible by pack factor

# after
--tp 2   # or tp 1

Defensive patterns

Strategy: validation

Validate before calling

inter, pack_num, tp = model_intermediate_size, 32 // weight_bits, tp_size
assert inter % (tp * pack_num) == 0, "output shard not packable; change tp_size"

Type guard

def tp_ok_for_output(out_size: int, pack_numerator: int, tp: int) -> bool:
    return out_size % tp == 0 and (out_size // tp) % pack_numerator == 0

Prevention

Use power-of-two TP degrees for GPTQ models
Check intermediate_size divisibility by tp * (32/weight_bits) before launching

When it happens

Trigger: sum(output_partition_sizes) % quant_config.pack_factor.numerator != 0, e.g. an intermediate size of 11008 with tp=3 yielding a shard not divisible by 8 (for 4-bit pack factor 8), or column-parallel layers sharded across a non-divisor TP degree.

Common situations: Same class as the K check: odd tensor-parallel degrees (3, 5, 6, 7) on models whose intermediate/hidden sizes are only divisible by 2/4/8; also fused or stacked projections with unusual output sizes.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/13edb6084aaebb65. Report an issue: GitHub.