{"record":{"id":"115d3148db92fe13","repo":"sgl-project/sglang","slug":"tensor-parallel-size-self-tp-size-is-greater-tha-115d31","errorCode":null,"errorMessage":"Tensor parallel size {self.tp_size} is greater than the number of experts {config.moe_num_experts}.","messagePattern":"Tensor parallel size (.+?) is greater than the number of experts (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/step3_vl.py","lineNumber":122,"sourceCode":"        x = self.act_fn(gate_up)\n        x, _ = self.down_proj(x)\n        return x\n\n\nclass Step3TextMoEMLP(nn.Module):\n    # Native\n    def __init__(\n        self,\n        layer_id: int,\n        config: Step3TextConfig,\n        quant_config: Optional[QuantizationConfig] = None,\n        prefix: str = \"\",\n    ):\n        super().__init__()\n        self.tp_size = get_parallel().tp_size\n        self.layer_id = layer_id\n        if self.tp_size > config.moe_num_experts:\n            raise ValueError(\n                f\"Tensor parallel size {self.tp_size} is greater than \"\n                f\"the number of experts {config.moe_num_experts}.\"\n            )\n\n        self.topk = TopK(\n            top_k=config.moe_top_k,\n            renormalize=config.norm_expert_weight,\n            use_grouped_topk=False,\n            layer_id=layer_id,\n        )\n\n        self.experts = get_moe_impl_class(quant_config)(\n            num_experts=config.moe_num_experts,\n            top_k=config.moe_top_k,\n            hidden_size=config.hidden_size,\n            intermediate_size=config.moe_intermediate_size,\n            layer_id=layer_id,\n            quant_config=quant_config,","sourceCodeStart":104,"sourceCodeEnd":140,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/step3_vl.py#L104-L140","documentation":"Step3-VL MoE experts are partitioned across TP ranks, so tp_size must not exceed config.moe_num_experts; otherwise some rank would own zero experts and weight loading would be undefined. Init fails fast with both numbers printed.","triggerScenarios":"Launching step3-vl with --tp-size > config.moe_num_experts (e.g. a small fine-grained MoE with 8 experts launched with tp=16).","commonSituations":"Reusing high-TP launch scripts from large MoE models; confusing moe_top_k with moe_num_experts when reasoning about parallelism limits.","solutions":["Lower --tp-size to <= moe_num_experts in config.json","Add --dp-size or --ep-size for extra parallelism instead of raising TP","Confirm the checkpoint's moe_num_experts value matches expectations"],"exampleFix":"# before\n--tp-size 16 --model step3-vl\n# after\n--tp-size 8 --dp-size 2 --model step3-vl","handlingStrategy":"validation","validationCode":"cfg = json.load(open(\"config.json\"))\nassert tp_size <= cfg[\"moe_num_experts\"], (tp_size, cfg[\"moe_num_experts\"])","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Cap TP by expert count; scale with DP/EP instead"],"tags":["step3-vl","tensor-parallel","moe"],"backgroundTag":"tp-size-exceeds-experts","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}