sgl-project/sglang · critical · ValueError

Tensor parallel size {self.tp_size} is greater than the numb

Error message

Tensor parallel size {self.tp_size} is greater than the number of experts {config.moe_num_experts}.

What it means

The MoE gate in Step3p5 requires at least one expert per tensor-parallel rank. If TP size exceeds moe_num_experts, expert weights cannot be sharded and the model raises during __init__.

Source

Thrown at python/sglang/srt/models/step3p5.py:131

        layer_id: int,
        quant_config: Optional[QuantizationConfig] = None,
        prefix: str = "",
    ):
        super().__init__()
        self.tp_size = get_parallel().tp_size
        self.layer_id = layer_id

        self.need_fp32_gate = config.need_fp32_gate
        self.routed_scaling_factor = config.moe_router_scaling_factor
        self.use_moe_router_bias = config.use_moe_router_bias
        if self.use_moe_router_bias:
            self.router_bias = nn.Parameter(
                torch.zeros(config.moe_num_experts, dtype=torch.float32),
                requires_grad=False,
            )

        if self.tp_size > config.moe_num_experts:
            raise ValueError(
                f"Tensor parallel size {self.tp_size} is greater than "
                f"the number of experts {config.moe_num_experts}."
            )

        self.limit = config.swiglu_limits[layer_id]
        self.limit = self.limit if self.limit > 0 else None

        self.topk = TopK(
            top_k=config.moe_top_k,
            renormalize=True,
            use_grouped_topk=False,
            scoring_func="sigmoid",
            correction_bias=self.router_bias,
            apply_routed_scaling_factor_on_output=False,
            layer_id=layer_id,
        )

        self.experts = get_moe_impl_class(quant_config)(

View on GitHub (pinned to 0132848349)

Solutions

1. Reduce --tensor-parallel-size to <= config.moe_num_experts
2. Check config.moe_num_experts in the model's config.json to confirm the true expert count
3. Use expert parallelism (--ep-size) instead if you need more GPUs than experts

Example fix

# before
python -m sglang.launch_server --model step3p5 --tp 8
# after (assuming 4 experts)
python -m sglang.launch_server --model step3p5 --tp 4

import json
cfg = json.load(open('config.json'))
tp = 4
assert tp <= cfg['moe_num_experts'], f"tp {tp} > experts {cfg['moe_num_experts']}"

When it happens

Trigger: Launching with --tp-size N where N > config.moe_num_experts (e.g. tp=8 on a 4-expert MoE layer).

Common situations: Reusing a large-TP launch command on a small dense/MoE variant; misreading expert count from config; A100/H100 multi-node scripts applied to a tiny test checkpoint.

Related errors

• LTX2Attention requires heads divisible by tp_size, got {self
• LTX2Attention requires inner_dim divisible by tp_size, got {
• out_channels must be divisible by tp_size for TP-sharded out
• MiniMax H3 attention heads must be divisible by TP size: {ar
• world_size ({world_size}) is not equal to tensor_model_paral

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/4a6cf6c8ef57085e. Report an issue: GitHub.