{"record":{"id":"6e5947ee90ad9d9e","repo":"sgl-project/sglang","slug":"unsupported-activation-activation-with-is-ga-6e5947","errorCode":null,"errorMessage":"Unsupported activation: {activation=}, with {is_gated=}","messagePattern":"Unsupported activation: (.+?), with (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe.py","lineNumber":805,"sourceCode":"        else:\n            if _has_vllm_ops:\n                vllm_ops.gelu_and_mul(\n                    intermediate_cache2, intermediate_cache1.view(-1, N)\n                )\n            else:\n                # Fallback: native PyTorch gelu_and_mul\n                x = intermediate_cache1.view(-1, N)\n                d = x.shape[-1] // 2\n                intermediate_cache2.copy_(F.gelu(x[..., :d]) * x[..., d:])\n    # Activation function without multiplication\n    elif activation == \"silu\" and not is_gated:\n        intermediate_cache2 = F.silu(intermediate_cache1.view(-1, N))\n    elif activation == \"gelu\" and not is_gated:\n        intermediate_cache2 = F.gelu(intermediate_cache1.view(-1, N))\n    elif activation == \"relu2\" and not is_gated:\n        intermediate_cache2 = torch.square(F.relu(intermediate_cache1.view(-1, N)))\n    else:\n        raise ValueError(f\"Unsupported activation: {activation=}, with {is_gated=}\")\n\n    del intermediate_cache1\n\n    intermediate_cache3 = torch.empty(\n        (num_tokens, topk, w2.shape[1]),\n        device=hidden_states.device,\n        dtype=hidden_states.dtype,\n    )\n\n    # LoRA hooks force the second kernel to write to intermediate_cache3 so\n    # hooks.after_down can inspect/modify it before reduction.\n    _use_intermediate = not no_combine and (topk != 1 or hooks)\n\n    out_slice = None\n    if use_fused_moe_sum_all_reduce:\n        out_slice = out_hidden_states\n        out_slice.zero_()\n","sourceCodeStart":787,"sourceCodeEnd":823,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe.py#L787-L823","documentation":"The non-Triton (torch/aten) fallback branch of the fused MoE kernel sequence only knows silu (gated), gelu (ungated) and relu2 (ungated) activations. Any other activation string, or a gated variant of gelu/relu2, reaches this ValueError. It is the catch-all guard for the reference implementation, not the fast kernels.","triggerScenarios":"Calling fused_experts_impl / the Triton runner's eager fallback with activation=\"gelu_tanh\", \"gelu\" while is_gated=True, or any custom activation string; small-token debugging paths that force the torch branch; models registering a novel activation for their MoE experts.","commonSituations":"Adding a new MoE model whose experts use an activation SGLang hasn't mapped (e.g. gated GELU variants), or a typo in an activation name in a config; running with the fallback kernel path on debug runs.","solutions":["Use one of the supported combos: silu+gated, gelu+ungated, relu2+ungated","If the model needs gated gelu, extend the branch in fused_moe.py:805 area (add `elif activation == \"gelu\" and is_gated: ...` computing gate/up product) and file an upstream PR","Check the model config's hidden_act / act_fn for typos or unmapped names and map it to a supported one"],"exampleFix":"# before\nfused_experts(..., activation=\"gelu\", is_gated=True)  # raises\n\n# after\nfused_experts(..., activation=\"silu\", is_gated=True)  # or add a gated-gelu branch","handlingStrategy":"validation","validationCode":"SUPPORTED_ACTIVATIONS = {(\"silu\", True), (\"gelu\", False), (\"relu2\", False)}\nassert (activation, bool(is_gated)) in SUPPORTED_ACTIVATIONS, \\\n    f\"unsupported activation {activation} gated={is_gated}\"","typeGuard":"def is_supported_activation(name: str, gated: bool) -> bool:\n    return (name, gated) in {(\"silu\", True), (\"gelu\", False), (\"relu2\", False)}","tryCatchPattern":null,"preventionTips":["Validate model hidden_act against the supported set when adding new MoE models","Run a 1-token smoke forward in CI to catch unsupported activation mapping early"],"tags":["moe","activation","fallback-kernel","validation"],"backgroundTag":"unsupported-activation-function","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}