sgl-project/sglang · error · ValueError

The input size is not aligned with the quantized weight shap

Error message

The input size is not aligned with the quantized weight shape.

What it means

bitsandbytes packs multiple elements per byte according to a dtype-dependent quant ratio; the total weight size (input_size_per_partition * output_size_per_partition) must be divisible by that ratio. If not, the packed uint8 buffer cannot be formed correctly.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/bitsandbytes.py:92

    def __init__(self, quant_config: BitsAndBytesConfig):
        require_bitsandbytes()
        self.quant_config = quant_config

    def create_weights(
        self,
        layer: torch.nn.Module,
        input_size_per_partition: int,
        output_partition_sizes: list[int],
        input_size: int,
        output_size: int,
        params_dtype: torch.dtype,
        **extra_weight_attrs,
    ) -> None:
        quant_ratio = calculate_quant_ratio(params_dtype)
        output_size_per_partition = sum(output_partition_sizes)
        total_size = input_size_per_partition * output_size_per_partition
        if total_size % quant_ratio != 0:
            raise ValueError(
                "The input size is not aligned with the quantized weight shape."
            )

        qweight = nn.Parameter(
            torch.empty(total_size // quant_ratio, 1, dtype=torch.uint8),
            requires_grad=False,
        )
        set_weight_attrs(
            qweight,
            {
                "input_dim": 0,
                "output_dim": 0,
                "pack_factor": quant_ratio,
                "use_bitsandbytes_4bit": True,
                "bnb_full_shape": (output_size, input_size),
                "bnb_local_shape": (
                    output_size_per_partition,
                    input_size_per_partition,

View on GitHub (pinned to 0132848349)

Solutions

1. Choose a tensor-parallel size that divides the dimension evenly and keeps total size aligned
2. Use a different quantization method for this model shape
3. Pad/adjust the layer shape if you control the model definition

from ...bitsandbytes import calculate_quant_ratio
if (input_size_per_partition * sum(output_partition_sizes)) % calculate_quant_ratio(params_dtype) != 0:
    raise SystemExit("layer size not aligned for bnb packing")

When it happens

Trigger: create_weights for a bnb 4-bit layer where in_features * out_features is not divisible by calculate_quant_ratio(params_dtype) — e.g. odd hidden sizes under tensor parallelism producing non-aligned partition sizes.

Common situations: TP sharding splitting a dimension into a size that breaks packing alignment; models with unusual (prime/odd) hidden dims quantized to 4-bit.

Related errors

• bitsandbytes 4-bit TP shard is not aligned to quantization b
• Shape mismatch: A K={k}, B K={b.shape[1] * 2}
• Parameter {param_name} not found in the model.
• bitsandbytes 4-bit TP only supports column-parallel output s
• bitsandbytes 4-bit TP does not support nested quant states.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/af30e8f19a45b8a1. Report an issue: GitHub.