sgl-project/sglang · error · ValueError

Weight input_size_per_partition = {input_size_per_partition}

Error message

Weight input_size_per_partition = {input_size_per_partition} is not divisible by weight quantization block_k = {block_k}.

What it means

Raised in create_weights of the FP8 quantization method when, under tensor parallelism with row-parallel weight sharding, the per-partition input dimension (input_size // tp_size) is not divisible by the FP8 blockwise weight block size (weight_block_size[1], i.e. block_k). Blockwise FP8 quantization requires each shard to align to quantization block boundaries.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/fp8.py:158

        output_partition_sizes: List[int],
        input_size: int,
        output_size: int,
        params_dtype: torch.dtype,
        **extra_weight_attrs,
    ):
        output_size_per_partition = sum(output_partition_sizes)
        weight_loader = extra_weight_attrs.get("weight_loader")

        tp_size = get_tensor_model_parallel_world_size()
        if self.block_quant:
            block_n, block_k = (
                self.quant_config.weight_block_size[0],
                self.quant_config.weight_block_size[1],
            )
            # Required by row parallel
            if tp_size > 1 and input_size // input_size_per_partition == tp_size:
                if input_size_per_partition % block_k != 0:
                    raise ValueError(
                        f"Weight input_size_per_partition = "
                        f"{input_size_per_partition} is not divisible by "
                        f"weight quantization block_k = {block_k}."
                    )
            # Required by column parallel or enabling merged weights
            if (
                tp_size > 1 and output_size // output_size_per_partition == tp_size
            ) or len(output_partition_sizes) > 1:
                for output_partition_size in output_partition_sizes:
                    if output_partition_size % block_n != 0:
                        raise ValueError(
                            f"Weight output_partition_size = "
                            f"{output_partition_size} is not divisible by "
                            f"weight quantization block_n = {block_n}."
                        )

        layer.logical_widths = output_partition_sizes
        layer.input_size_per_partition = input_size_per_partition

View on GitHub (pinned to 0132848349)

Solutions

1. Pick a tensor-parallel size where input_size/tp_size is divisible by weight_block_size[1] (usually 128) — e.g. reduce tp_size until aligned
2. Check the model config's hidden/intermediate size for blockwise-FP8 compatibility before launching
3. Use a non-blockwise FP8 scheme (per-tensor scales) if the dimension can't be aligned
4. Fall back to tp_size=1 for the offending layer group or use a different quantization method

Example fix

# before
--tensor-parallel-size 6   # 4608/6=768 ok, but 3000/6=500 not % 128
# after
--tensor-parallel-size 2   # ensure input_size/tp % block_k == 0
# or validate up front:
assert (hidden // tp) % quant_config.weight_block_size[1] == 0

block_k = quant_config.weight_block_size[1]
for layer in linear_layers:
    if tp_size > 1 and layer.input_size % tp_size == 0:
        assert (layer.input_size // tp_size) % block_k == 0, f'{layer.input_size}/{tp_size} not % {block_k}'

When it happens

Trigger: Creating FP8 blockwise-quantized weights with tp_size>1 where input_size (in_features) / tp_size leaves a remainder modulo weight_block_size[1] (commonly 128); e.g. in_features=6144 with tp=8 gives 768 which is fine, but in_features=3000 with tp=2 gives 1500, not divisible by 128.

Common situations: Choosing a TP degree that doesn't evenly split hidden dims into 128-blocks, loading a model whose intermediate size isn't block-aligned, or overriding --quantization fp8 with --weight-block-size on a model not designed for blockwise FP8.

Related errors

• Weight output_partition_size = {output_partition_size} is no
• The output_size of gate's and up's weight = {intermediate_si
• The input_size of down's weight = {intermediate_size_per_par
• Weight input_size_per_partition = {input_size_per_partition}
• Weight output_partition_size = {output_partition_size} is no

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/70334a228c5a0a4e. Report an issue: GitHub.