sgl-project/sglang · error · ValueError

Qwen3-Next MTP shared expert fusion currently supports exact

Error message

Qwen3-Next MTP shared expert fusion currently supports exactly one shared expert because checkpoint weight remapping maps it into a single fused MoE expert slot.

What it means

The Qwen3-Next MTP (multi-token prediction) draft model applies the same constraint: shared-expert fusion supports exactly one shared expert, since load_weights remaps mlp.shared_expert.* into a single fused MoE expert slot.

Source

Thrown at python/sglang/srt/models/qwen3_next_mtp.py:93

            prefix=add_prefix("model", prefix),
            is_nextn=True,
        )
        self.lm_head = ParallelLMHead(
            config.vocab_size,
            config.hidden_size,
            quant_config=quant_config,
            prefix=add_prefix("model.shared_head.head", prefix),
            use_attn_tp_group=get_parallel().enable_dp_lm_head,
        )
        self.logits_processor = LogitsProcessor(config)
        # Mirror Qwen3NextForCausalLM.__init__'s shared-expert fusion setup so
        # the inherited load_weights() can find the attribute on the MTP path.
        # We compute it from the actual MTP MoE layer (1 layer with is_nextn=True),
        # not hardcode it — when the layer's MoE pre-fuses the shared expert,
        # load_weights must remap mlp.shared_expert.* into the fused slot.
        self.num_fused_shared_experts = self._get_num_fused_shared_experts()
        if self.num_fused_shared_experts > 1:
            raise ValueError(
                "Qwen3-Next MTP shared expert fusion currently supports exactly one "
                "shared expert because checkpoint weight remapping maps it into "
                "a single fused MoE expert slot."
            )
        self.enable_shared_expert_fusion = self.num_fused_shared_experts > 0

    @torch.no_grad()
    def forward(
        self,
        input_ids: torch.Tensor,
        positions: torch.Tensor,
        forward_batch: ForwardBatch,
        input_embeds: Optional[torch.Tensor] = None,
        **kwargs,
    ):
        exit_stack = ExitStack()
        if (
            is_npu()

View on GitHub (pinned to 0132848349)

Solutions

Use the official MTP checkpoint config
Disable shared expert fusion for the MTP layer
Verify shared_expert_intermediate_size / intermediate_size / num_experts consistency

Defensive patterns

Strategy: validation

Validate before calling

fused = mtp_moe._get_num_fused_shared_experts()
assert fused <= 1, f"{fused} fused shared experts unsupported"

Prevention

Use official MTP checkpoint configs
Validate shared expert sizing before enabling MTP

When it happens

Trigger: Initializing the Qwen3-Next MTP module where the MTP MoE layer's _get_num_fused_shared_experts() > 1, e.g. mismatched shared_expert_intermediate_size vs per-expert size.

Common situations: Serving Qwen3-Next with MTP speculative decoding using a custom config or a checkpoint with multiple shared experts.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6dd8d71ca5663acd. Report an issue: GitHub.