sgl-project/sglang · error · ValueError

Tensor parallel size {self.tp_size} is greater than the numb

Error message

Tensor parallel size {self.tp_size} is greater than the number of experts {config.moe_num_experts}.

What it means

Step3-VL MoE experts are partitioned across TP ranks, so tp_size must not exceed config.moe_num_experts; otherwise some rank would own zero experts and weight loading would be undefined. Init fails fast with both numbers printed.

Source

Thrown at python/sglang/srt/models/step3_vl.py:122

        x = self.act_fn(gate_up)
        x, _ = self.down_proj(x)
        return x


class Step3TextMoEMLP(nn.Module):
    # Native
    def __init__(
        self,
        layer_id: int,
        config: Step3TextConfig,
        quant_config: Optional[QuantizationConfig] = None,
        prefix: str = "",
    ):
        super().__init__()
        self.tp_size = get_parallel().tp_size
        self.layer_id = layer_id
        if self.tp_size > config.moe_num_experts:
            raise ValueError(
                f"Tensor parallel size {self.tp_size} is greater than "
                f"the number of experts {config.moe_num_experts}."
            )

        self.topk = TopK(
            top_k=config.moe_top_k,
            renormalize=config.norm_expert_weight,
            use_grouped_topk=False,
            layer_id=layer_id,
        )

        self.experts = get_moe_impl_class(quant_config)(
            num_experts=config.moe_num_experts,
            top_k=config.moe_top_k,
            hidden_size=config.hidden_size,
            intermediate_size=config.moe_intermediate_size,
            layer_id=layer_id,
            quant_config=quant_config,

View on GitHub (pinned to 0132848349)

Solutions

1. Lower --tp-size to <= moe_num_experts in config.json
2. Add --dp-size or --ep-size for extra parallelism instead of raising TP
3. Confirm the checkpoint's moe_num_experts value matches expectations

Example fix

# before
--tp-size 16 --model step3-vl
# after
--tp-size 8 --dp-size 2 --model step3-vl

cfg = json.load(open("config.json"))
assert tp_size <= cfg["moe_num_experts"], (tp_size, cfg["moe_num_experts"])

When it happens

Trigger: Launching step3-vl with --tp-size > config.moe_num_experts (e.g. a small fine-grained MoE with 8 experts launched with tp=16).

Common situations: Reusing high-TP launch scripts from large MoE models; confusing moe_top_k with moe_num_experts when reasoning about parallelism limits.

Related errors

• The output_size of gate's and up's weight = {intermediate_si
• The input_size of down's weight = {intermediate_size_per_par
• expert-pack v1 supports only single-GPU TP=EP=1
• The output_size of gate's and up's weight = {intermediate_si
• The input_size of down's weight = {intermediate_size_per_par

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/115d3148db92fe13. Report an issue: GitHub.