{"record":{"id":"6dd8d71ca5663acd","repo":"sgl-project/sglang","slug":"qwen3-next-mtp-shared-expert-fusion-currently-supp","errorCode":null,"errorMessage":"Qwen3-Next MTP shared expert fusion currently supports exactly one shared expert because checkpoint weight remapping maps it into a single fused MoE expert slot.","messagePattern":"Qwen3-Next MTP shared expert fusion currently supports exactly one shared expert because checkpoint weight remapping maps it into a single fused MoE expert slot\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/qwen3_next_mtp.py","lineNumber":93,"sourceCode":"            prefix=add_prefix(\"model\", prefix),\n            is_nextn=True,\n        )\n        self.lm_head = ParallelLMHead(\n            config.vocab_size,\n            config.hidden_size,\n            quant_config=quant_config,\n            prefix=add_prefix(\"model.shared_head.head\", prefix),\n            use_attn_tp_group=get_parallel().enable_dp_lm_head,\n        )\n        self.logits_processor = LogitsProcessor(config)\n        # Mirror Qwen3NextForCausalLM.__init__'s shared-expert fusion setup so\n        # the inherited load_weights() can find the attribute on the MTP path.\n        # We compute it from the actual MTP MoE layer (1 layer with is_nextn=True),\n        # not hardcode it — when the layer's MoE pre-fuses the shared expert,\n        # load_weights must remap mlp.shared_expert.* into the fused slot.\n        self.num_fused_shared_experts = self._get_num_fused_shared_experts()\n        if self.num_fused_shared_experts > 1:\n            raise ValueError(\n                \"Qwen3-Next MTP shared expert fusion currently supports exactly one \"\n                \"shared expert because checkpoint weight remapping maps it into \"\n                \"a single fused MoE expert slot.\"\n            )\n        self.enable_shared_expert_fusion = self.num_fused_shared_experts > 0\n\n    @torch.no_grad()\n    def forward(\n        self,\n        input_ids: torch.Tensor,\n        positions: torch.Tensor,\n        forward_batch: ForwardBatch,\n        input_embeds: Optional[torch.Tensor] = None,\n        **kwargs,\n    ):\n        exit_stack = ExitStack()\n        if (\n            is_npu()","sourceCodeStart":75,"sourceCodeEnd":111,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/qwen3_next_mtp.py#L75-L111","documentation":"The Qwen3-Next MTP (multi-token prediction) draft model applies the same constraint: shared-expert fusion supports exactly one shared expert, since load_weights remaps mlp.shared_expert.* into a single fused MoE expert slot.","triggerScenarios":"Initializing the Qwen3-Next MTP module where the MTP MoE layer's _get_num_fused_shared_experts() > 1, e.g. mismatched shared_expert_intermediate_size vs per-expert size.","commonSituations":"Serving Qwen3-Next with MTP speculative decoding using a custom config or a checkpoint with multiple shared experts.","solutions":["Use the official MTP checkpoint config","Disable shared expert fusion for the MTP layer","Verify shared_expert_intermediate_size / intermediate_size / num_experts consistency"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"fused = mtp_moe._get_num_fused_shared_experts()\nassert fused <= 1, f\"{fused} fused shared experts unsupported\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use official MTP checkpoint configs","Validate shared expert sizing before enabling MTP"],"tags":["qwen3-next","mtp","shared-expert","speculative-decoding"],"backgroundTag":"unsupported-model-config","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}