sgl-project/sglang · error · ValueError

Qwen3-Next shared expert fusion currently supports exactly o

Error message

Qwen3-Next shared expert fusion currently supports exactly one shared expert because checkpoint weight remapping maps it into a single fused MoE expert slot.

What it means

Qwen3-Next can fuse the shared (always-on) expert into one fused MoE expert slot during checkpoint remapping; the implementation only supports exactly one shared expert. _get_num_fused_shared_experts() returning >1 is rejected.

Source

Thrown at python/sglang/srt/models/qwen3_next.py:1038

        self.quant_config = quant_config
        self.model = Qwen3NextModel(
            config, quant_config, prefix=add_prefix("model", prefix)
        )
        self.lm_head = ParallelLMHead(
            config.vocab_size,
            config.hidden_size,
            quant_config=quant_config,
            org_num_embeddings=config.vocab_size,
            prefix=add_prefix("lm_head", prefix),
            use_attn_tp_group=get_parallel().enable_dp_lm_head,
        )
        self.logits_processor = LogitsProcessor(config)
        # For EAGLE3 support
        self.capture_aux_hidden_states = False

        self.num_fused_shared_experts = self._get_num_fused_shared_experts()
        if self.num_fused_shared_experts > 1:
            raise ValueError(
                "Qwen3-Next shared expert fusion currently supports exactly one "
                "shared expert because checkpoint weight remapping maps it into "
                "a single fused MoE expert slot."
            )
        self.enable_shared_expert_fusion = self.num_fused_shared_experts > 0

        self._routed_experts_weights_of_layer = LazyValue(
            lambda: {
                layer_id: layer.mlp.get_moe_weights()
                for layer_id, layer in enumerate(self.model.layers)
                if isinstance(layer.mlp, Qwen2MoeSparseMoeBlock)
            }
        )

    @property
    def routed_experts_weights_of_layer(self):
        return self._routed_experts_weights_of_layer.value

View on GitHub (pinned to 0132848349)

Solutions

1. Use the official Qwen3-Next config with a single shared expert
2. Set num_fused_shared_experts to 0 (disable shared expert fusion) via the corresponding server arg/config override
3. Verify shared_expert_intermediate_size and intermediate_size in config.json match the released checkpoint

Example fix

# before
config.shared_expert_intermediate_size = 2 * (config.intermediate_size // config.num_experts)
# after
config.shared_expert_intermediate_size = config.intermediate_size // config.num_experts  # exactly one fused slot

n = intermediate_size // (shared_expert_intermediate_size * 0 + 1)  # see _get_num_fused_shared_experts
# simpler: verify shared size fits exactly one expert slot
assert config.shared_expert_intermediate_size * config.num_experts <= config.intermediate_size

When it happens

Trigger: A Qwen3-Next checkpoint/config where shared_expert_intermediate_size divides intermediate_size such that more than one shared expert would be fused (custom or future checkpoints).

Common situations: Using a modified/community Qwen3-Next config with multiple shared experts or altered intermediate sizes.

Related errors

• Qwen3-Next MTP shared expert fusion currently supports exact
• DeepEP v2 MoE has not validated fused shared experts yet. Re
• num_token_non_padded must be a torch.Tensor
• num_token_non_padded must be a single-element tensor, got sh
• num_token_non_padded must be an integer tensor, got {num_tok

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/31de71d8930f2d20. Report an issue: GitHub.