sgl-project/sglang · error · ValueError

The input_size of down's weight = {intermediate_size_per_par

Error message

The input_size of down's weight = {intermediate_size_per_partition} is not divisible by weight quantization block_k = {block_k}.

What it means

Companion check for MoE down-projection: with TP>1 the down layer's input size (= sharded intermediate size) must be divisible by block_k so FP8 block scales align on the K axis. Raised in create_fp8_moe_weight_ at model init.

Source

Thrown at python/sglang/srt/layers/quantization/fp8.py:1178

            block_n, block_k = (
                quant_config.weight_block_size[0],
                quant_config.weight_block_size[1],
            )

            padding_size = get_moe_padding_size(_use_aiter)
            if not (_use_aiter and padding_size == block_n == block_k):
                # NOTE(HandH1998): To ensure proper alignment of the block-wise quantization scales, the output_size of the weights for both the gate and up layers must be divisible by block_n.
                # Required by column parallel or enabling merged weights
                if intermediate_size_per_partition % block_n != 0:
                    raise ValueError(
                        f"The output_size of gate's and up's weight = "
                        f"{intermediate_size_per_partition} is not divisible by "
                        f"weight quantization block_n = {block_n}."
                    )
                if tp_size > 1:
                    # Required by row parallel
                    if intermediate_size_per_partition % block_k != 0:
                        raise ValueError(
                            f"The input_size of down's weight = "
                            f"{intermediate_size_per_partition} is not divisible by "
                            f"weight quantization block_k = {block_k}."
                        )

        # WEIGHTS
        if is_fp4_expert:
            w13_weight = torch.nn.Parameter(
                torch.empty(
                    num_experts,
                    w13_num_shards * intermediate_size_per_partition,
                    hidden_size // 2,
                    dtype=torch.int8,
                ),
                requires_grad=False,
            )
            w2_weight = torch.nn.Parameter(
                torch.empty(

View on GitHub (pinned to 0132848349)

Solutions

1. Use a TP size making moe_intermediate_size/tp divisible by 128 (compute before launch)
2. Fall back to TP=1 (may need more memory) or a compatible TP
3. Choose a checkpoint without block quantization if flexible

Example fix

# before
--tp 5  # down input partition misaligned
# after
--tp 4

inter = model_config.moe_intermediate_size
assert args.tp == 1 or (inter // args.tp) % 128 == 0, "down-proj K partition misaligned; adjust --tp-size"

def down_proj_tp_ok(inter_size: int, tp: int, block_k: int = 128) -> bool:
    return tp == 1 or (inter_size // tp) % block_k == 0

When it happens

Trigger: FP8 block-quant MoE with tp_size>1 and moe_intermediate_size/tp not a multiple of block_k (128); typically the same misconfigured TP size that trips the gate/up check.

Common situations: Odd TP sizes on FP8 MoE models; models whose moe_intermediate_size has few valid TP factorizations.

Related errors

• The output_size of gate's and up's weight = {intermediate_si
• The output_size of gate's and up's weight = {intermediate_si
• The input_size of down's weight = {intermediate_size_per_par
• Weight input_size_per_partition = {input_size_per_partition}
• Weight output_partition_size = {output_partition_size} is no

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8795f89a4d802520. Report an issue: GitHub.