sgl-project/sglang · error · ValueError

The output_size of gate's and up's weight = {intermediate_si

Error message

The output_size of gate's and up's weight = {intermediate_size_per_partition} is not divisible by weight quantization block_n = {block_n}.

What it means

For block-quant FP8 MoE layers (unless on ROCm AIter with matching padding), the per-partition intermediate size (gate/up output size) must be divisible by block_n so FP8 block scale rows align. create_fp8_moe_weight_ raises during weight creation, i.e. at model init.

Source

Thrown at python/sglang/srt/layers/quantization/fp8.py:1170

        w13_up_dim, w2_up_dim, weight_padded = get_moe_weight_sizes(
            intermediate_size_per_partition,
            is_aiter_moe=_use_aiter,
            is_concat=layer.moe_runner_config.is_gated,
            is_packed=False,
        )

        if block_quant:
            block_n, block_k = (
                quant_config.weight_block_size[0],
                quant_config.weight_block_size[1],
            )

            padding_size = get_moe_padding_size(_use_aiter)
            if not (_use_aiter and padding_size == block_n == block_k):
                # NOTE(HandH1998): To ensure proper alignment of the block-wise quantization scales, the output_size of the weights for both the gate and up layers must be divisible by block_n.
                # Required by column parallel or enabling merged weights
                if intermediate_size_per_partition % block_n != 0:
                    raise ValueError(
                        f"The output_size of gate's and up's weight = "
                        f"{intermediate_size_per_partition} is not divisible by "
                        f"weight quantization block_n = {block_n}."
                    )
                if tp_size > 1:
                    # Required by row parallel
                    if intermediate_size_per_partition % block_k != 0:
                        raise ValueError(
                            f"The input_size of down's weight = "
                            f"{intermediate_size_per_partition} is not divisible by "
                            f"weight quantization block_k = {block_k}."
                        )

        # WEIGHTS
        if is_fp4_expert:
            w13_weight = torch.nn.Parameter(
                torch.empty(
                    num_experts,

View on GitHub (pinned to 0132848349)

Solutions

1. Pick a TP size where moe_intermediate_size/tp is a multiple of 128 (often TP=1,2,4,8)
2. Check the model's moe_intermediate_size and compute divisibility before launching
3. Use a non-block-quant checkpoint or an AIter path on ROCm gfx95 if applicable

Example fix

# before
--tp 6  # 1536/6 = 256 ok, but e.g. 1408/6 not multiple of 128
# after
--tp 2  # intermediate_size_per_partition % 128 == 0

inter = model_config.moe_intermediate_size; block_n = 128; tp = args.tp
assert tp == 1 or (inter // tp) % block_n == 0, f"moe intermediate/tp={inter//tp} not divisible by {block_n}"

def moe_tp_ok(inter_size: int, tp: int, block: int = 128) -> bool:
    return tp == 1 or (inter_size // tp) % block == 0

When it happens

Trigger: Creating FP8 MoE weights with intermediate_size_per_partition = intermediate_size / tp not divisible by block_n (128) — e.g. moe_intermediate_size 1408 with TP=2 giving 704, not a multiple of 128.

Common situations: Running FP8 MoE models (DeepSeek, Qwen-MoE, Mixtral-FP8) with TP sizes that shard intermediate size into non-128 multiples; models with small moe_intermediate_size.

Related errors

• The input_size of down's weight = {intermediate_size_per_par
• The output_size of gate's and up's weight = {intermediate_si
• The input_size of down's weight = {intermediate_size_per_par
• Weight input_size_per_partition = {input_size_per_partition}
• Weight output_partition_size = {output_partition_size} is no

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/81e7ef50eb196c32. Report an issue: GitHub.