sgl-project/sglang · error · ValueError

Weight input_size_per_partition = {input_size_per_partition}

Error message

Weight input_size_per_partition = {input_size_per_partition} is not divisible by weight quantization block_k = {block_k}.

What it means

validate_block_quant_shapes enforces that for row-parallel (input-sharded) linear layers under tensor parallelism, the per-partition input dimension must be divisible by the weight quantization block_k (e.g. 128). Otherwise FP8 block scale tiles would straddle partition boundaries and produce wrong results.

Source

Thrown at python/sglang/srt/layers/quantization/fp8.py:508

        output_size_per_partition: int,
        output_partition_sizes: List[int],
        skip_block_quant_check: bool = False,
    ):
        block_n, block_k = (
            quant_config.weight_block_size[0],
            quant_config.weight_block_size[1],
        )

        if skip_block_quant_check:
            print_warning_once(
                "Skipping block quantization checks for weight partition."
            )
        else:
            tp_size = get_parallel().tp_size
            # Required by row parallel
            if tp_size > 1 and input_size // input_size_per_partition == tp_size:
                if input_size_per_partition % block_k != 0:
                    raise ValueError(
                        f"Weight input_size_per_partition = "
                        f"{input_size_per_partition} is not divisible by "
                        f"weight quantization block_k = {block_k}."
                    )
            # Required by column parallel or enabling merged weights
            if (
                tp_size > 1 and output_size // output_size_per_partition == tp_size
            ) or len(output_partition_sizes) > 1:
                for output_partition_size in output_partition_sizes:
                    if output_partition_size % block_n != 0:
                        raise ValueError(
                            f"Weight output_partition_size = "
                            f"{output_partition_size} is not divisible by "
                            f"weight quantization block_n = {block_n}."
                        )

    @staticmethod
    def create_fp8_weight_(

View on GitHub (pinned to 0132848349)

Solutions

1. Use a tensor-parallel size that divides the hidden/input size into a multiple of block_k (128), e.g. TP in powers of two for standard models
2. Reduce --tp-size (e.g. to 2 or 1) so input_size/tp is a multiple of 128
3. If the model architecture allows, pad or choose a checkpoint variant whose K dim is TP-friendly

Example fix

# before
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3 --tp 6  # K partition % 128 != 0
# after
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3 --tp 8

tp = 4; block_k = 128; hidden = model_config.hidden_size
assert tp == 1 or (hidden // tp) % block_k == 0, f"hidden/tp={hidden//tp} not divisible by {block_k}; pick another --tp-size"

def tp_ok_for_block_k(input_size: int, tp: int, block_k: int = 128) -> bool:
    return tp == 1 or (input_size // tp) % block_k == 0

When it happens

Trigger: Creating an FP8 block-quant linear layer with tp_size>1 where the K dimension divided by TP is not a multiple of block_k — e.g. hidden_size 4096 with TP=3 giving input_size_per_partition 1366 with block_k=128, or small models whose hidden size isn't TP-divisible into 128-multiples.

Common situations: Odd tensor-parallel sizes (3, 5, 6) on block-quant FP8 models; small custom models with hidden_size not divisible by TP*128; DeepSeek FP8 with unusual --tp-size.

Related errors

• Weight output_partition_size = {output_partition_size} is no
• The output_size of gate's and up's weight = {intermediate_si
• The input_size of down's weight = {intermediate_size_per_par
• Weight input_size_per_partition = {input_size_per_partition}
• Weight output_partition_size = {output_partition_size} is no

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/616156ed99789503. Report an issue: GitHub.