{"record":{"id":"31de71d8930f2d20","repo":"sgl-project/sglang","slug":"qwen3-next-shared-expert-fusion-currently-supports","errorCode":null,"errorMessage":"Qwen3-Next shared expert fusion currently supports exactly one shared expert because checkpoint weight remapping maps it into a single fused MoE expert slot.","messagePattern":"Qwen3-Next shared expert fusion currently supports exactly one shared expert because checkpoint weight remapping maps it into a single fused MoE expert slot\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/qwen3_next.py","lineNumber":1038,"sourceCode":"        self.quant_config = quant_config\n        self.model = Qwen3NextModel(\n            config, quant_config, prefix=add_prefix(\"model\", prefix)\n        )\n        self.lm_head = ParallelLMHead(\n            config.vocab_size,\n            config.hidden_size,\n            quant_config=quant_config,\n            org_num_embeddings=config.vocab_size,\n            prefix=add_prefix(\"lm_head\", prefix),\n            use_attn_tp_group=get_parallel().enable_dp_lm_head,\n        )\n        self.logits_processor = LogitsProcessor(config)\n        # For EAGLE3 support\n        self.capture_aux_hidden_states = False\n\n        self.num_fused_shared_experts = self._get_num_fused_shared_experts()\n        if self.num_fused_shared_experts > 1:\n            raise ValueError(\n                \"Qwen3-Next shared expert fusion currently supports exactly one \"\n                \"shared expert because checkpoint weight remapping maps it into \"\n                \"a single fused MoE expert slot.\"\n            )\n        self.enable_shared_expert_fusion = self.num_fused_shared_experts > 0\n\n        self._routed_experts_weights_of_layer = LazyValue(\n            lambda: {\n                layer_id: layer.mlp.get_moe_weights()\n                for layer_id, layer in enumerate(self.model.layers)\n                if isinstance(layer.mlp, Qwen2MoeSparseMoeBlock)\n            }\n        )\n\n    @property\n    def routed_experts_weights_of_layer(self):\n        return self._routed_experts_weights_of_layer.value\n","sourceCodeStart":1020,"sourceCodeEnd":1056,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/qwen3_next.py#L1020-L1056","documentation":"Qwen3-Next can fuse the shared (always-on) expert into one fused MoE expert slot during checkpoint remapping; the implementation only supports exactly one shared expert. _get_num_fused_shared_experts() returning >1 is rejected.","triggerScenarios":"A Qwen3-Next checkpoint/config where shared_expert_intermediate_size divides intermediate_size such that more than one shared expert would be fused (custom or future checkpoints).","commonSituations":"Using a modified/community Qwen3-Next config with multiple shared experts or altered intermediate sizes.","solutions":["Use the official Qwen3-Next config with a single shared expert","Set num_fused_shared_experts to 0 (disable shared expert fusion) via the corresponding server arg/config override","Verify shared_expert_intermediate_size and intermediate_size in config.json match the released checkpoint"],"exampleFix":"# before\nconfig.shared_expert_intermediate_size = 2 * (config.intermediate_size // config.num_experts)\n# after\nconfig.shared_expert_intermediate_size = config.intermediate_size // config.num_experts  # exactly one fused slot","handlingStrategy":"validation","validationCode":"n = intermediate_size // (shared_expert_intermediate_size * 0 + 1)  # see _get_num_fused_shared_experts\n# simpler: verify shared size fits exactly one expert slot\nassert config.shared_expert_intermediate_size * config.num_experts <= config.intermediate_size","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use official Qwen3-Next configs","Diff custom configs against upstream before loading"],"tags":["qwen3-next","moe","shared-expert","fusion"],"backgroundTag":"unsupported-model-config","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}