sgl-project/sglang · error · ValueError

The output_size of gate's and up's weight = {intermediate_si

Error message

The output_size of gate's and up's weight = {intermediate_size_per_partition} is not divisible by weight quantization block_n = {block_n}.

What it means

When using block-quantized (grouped) FP8 MoE weights, the intermediate size per partition must be divisible by the weight block_n so scale tensors stay aligned after column-parallel sharding or merged gate/up projections. This raises when intermediate_size_per_partition % block_n != 0.

Source

Thrown at python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8_moe.py:112

    ):
        from sglang.srt.layers.moe.fused_moe_triton import FusedMoeWeightScaleSupported

        params_dtype = torch.float8_e4m3fn

        if self.block_quant:
            assert self.weight_block_size is not None
            layer.weight_block_size = self.weight_block_size
            tp_size = get_parallel().tp_size
            block_n, block_k = (
                self.weight_block_size[0],
                self.weight_block_size[1],
            )
            # NOTE: To ensure proper alignment of the block-wise quantization
            # scales, the output_size of the weights for both the gate and up
            # layers must be divisible by block_n.
            # Required by column parallel or enabling merged weights
            if intermediate_size_per_partition % block_n != 0:
                raise ValueError(
                    f"The output_size of gate's and up's weight = "
                    f"{intermediate_size_per_partition} is not divisible by "
                    f"weight quantization block_n = {block_n}."
                )
            if tp_size > 1 and intermediate_size_per_partition % block_k != 0:
                # Required by row parallel
                raise ValueError(
                    f"The input_size of down's weight = "
                    f"{intermediate_size_per_partition} is not divisible by "
                    f"weight quantization block_k = {block_k}."
                )

        w13_up_dim, w2_down_dim, weight_padded = get_moe_weight_sizes(
            intermediate_size_per_partition,
            is_aiter_moe=_use_aiter,
            is_concat=True,
            is_packed=False,
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Change tensor-parallel size so intermediate_size_per_partition is divisible by block_n
2. Use a checkpoint whose block size divides the intermediate size
3. Disable TP for this model if its geometry can't be sharded evenly

Example fix

# before: intermediate_size=5504, block_n=128, --tp 8
# after: choose --tp 4 so 2*5504/4 is divisible by 128, or use a compatible checkpoint

inter = model_config.intermediate_size; tp = 8; block_n = cfg["quantization_config"]["weights"].get("block_size", [128,128])[1]
assert (2*inter//tp) % block_n == 0, "intermediate size per partition not divisible by block_n"

When it happens

Trigger: Loading a block-quantized FP8 MoE model where (2 * intermediate_size / tp_size) is not divisible by block_n (e.g. block_n=128 with an odd intermediate size or a TP degree that shards it into non-multiples).

Common situations: DeepSeek-style FP8 block-quant models launched with an unusual --tp size; models with intermediate_size not a multiple of the quant block size.

Related errors

• The input_size of down's weight = {intermediate_size_per_par
• Weight output_partition_size = {output_partition_size} is no
• The output_size of gate's and up's weight = {intermediate_si
• The input_size of down's weight = {intermediate_size_per_par
• Only block_quant=True is supported in Quark MXFP4 requantiza

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/77b2095aa6da2cf8. Report an issue: GitHub.