{"record":{"id":"7cb493c699fed211","repo":"Comfy-Org/ComfyUI","slug":"insupportable-scoring-function-for-moe-gating-se","errorCode":null,"errorMessage":"insupportable scoring function for MoE gating: {self.scoring_func}","messagePattern":"insupportable scoring function for MoE gating: (.+?)","errorType":"validation","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"comfy/ldm/hidream/model.py","lineNumber":272,"sourceCode":"        self.gating_dim = embed_dim\n        self.weight = nn.Parameter(torch.empty((self.n_routed_experts, self.gating_dim), dtype=dtype, device=device))\n        self.reset_parameters()\n\n    def reset_parameters(self) -> None:\n        pass\n        # import torch.nn.init  as init\n        # init.kaiming_uniform_(self.weight, a=math.sqrt(5))\n\n    def forward(self, hidden_states):\n        bsz, seq_len, h = hidden_states.shape\n\n        ### compute gating score\n        hidden_states = hidden_states.view(-1, h)\n        logits = F.linear(hidden_states, comfy.model_management.cast_to(self.weight, dtype=hidden_states.dtype, device=hidden_states.device), None)\n        if self.scoring_func == 'softmax':\n            scores = logits.softmax(dim=-1)\n        else:\n            raise NotImplementedError(f'insupportable scoring function for MoE gating: {self.scoring_func}')\n\n        ### select top-k experts\n        topk_weight, topk_idx = torch.topk(scores, k=self.top_k, dim=-1, sorted=False)\n\n        ### norm gate to sum 1\n        if self.top_k > 1 and self.norm_topk_prob:\n            denominator = topk_weight.sum(dim=-1, keepdim=True) + 1e-20\n            topk_weight = topk_weight / denominator\n\n        aux_loss = None\n        return topk_idx, topk_weight, aux_loss\n\n\n# Modified from https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py\nclass MOEFeedForwardSwiGLU(nn.Module):\n    def __init__(\n        self,\n        dim: int,","sourceCodeStart":254,"sourceCodeEnd":290,"githubUrl":"https://github.com/Comfy-Org/ComfyUI/blob/1c6d8d45b3693bfbb32385b410d813a7fd6be216/comfy/ldm/hidream/model.py#L254-L290","documentation":"HiDream's MoE router computes gating scores and only implements scoring_func == 'softmax'; anything else raises NotImplementedError. The attribute comes from the checkpoint config of HiDream's MoE LLM layers (based on Qwen/LLaVA-style MoE routing).","triggerScenarios":"Loading a HiDream checkpoint whose MoE config sets a different scoring function (e.g. 'sigmoid' as used in some Qwen3/Llama-4 style routers), or constructing the gate module manually with a non-softmax string.","commonSituations":"HiDream checkpoints built on newer MoE architectures that switched gating to sigmoid scoring; community merges of config files from different model generations.","solutions":["Use a HiDream checkpoint whose MoE config specifies scoring_func='softmax' (the release models).","If porting a sigmoid-router variant, add a branch: scores = logits.sigmoid() when scoring_func == 'sigmoid', matching the source model's normalization.","Verify the config key is being read from the right section (per-layer MoE config, not global model config)."],"exampleFix":"# before (config)\n\"scoring_func\": \"sigmoid\"\n\n# after (config)\n\"scoring_func\": \"softmax\"\n\n# or, if the checkpoint truly uses sigmoid routing, extend forward():\nif self.scoring_func == 'softmax':\n    scores = logits.softmax(dim=-1)\nelif self.scoring_func == 'sigmoid':\n    scores = logits.sigmoid()","handlingStrategy":"validation","validationCode":"assert layer.scoring_func == \"softmax\", f\"MoE gate scoring_func {layer.scoring_func!r} is not implemented\"","typeGuard":"def is_softmax_router(scoring_func: str) -> bool:\n    return scoring_func == \"softmax\"","tryCatchPattern":null,"preventionTips":["Inspect the MoE section of the checkpoint config before loading HiDream variants.","When porting sigmoid-router architectures, extend the gate forward in the same change."],"tags":["hidream","moe","not-implemented","config"],"backgroundTag":null,"analyzedSha":"1c6d8d45b3693bfbb32385b410d813a7fd6be216","analyzedAt":"2026-08-14T19:37:18.893Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}