sgl-project/sglang · error · ValueError

Unsupported activation: {activation=}, with {is_gated=}

Error message

Unsupported activation: {activation=}, with {is_gated=}

What it means

The non-Triton (torch/aten) fallback branch of the fused MoE kernel sequence only knows silu (gated), gelu (ungated) and relu2 (ungated) activations. Any other activation string, or a gated variant of gelu/relu2, reaches this ValueError. It is the catch-all guard for the reference implementation, not the fast kernels.

Source

Thrown at python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe.py:805

        else:
            if _has_vllm_ops:
                vllm_ops.gelu_and_mul(
                    intermediate_cache2, intermediate_cache1.view(-1, N)
                )
            else:
                # Fallback: native PyTorch gelu_and_mul
                x = intermediate_cache1.view(-1, N)
                d = x.shape[-1] // 2
                intermediate_cache2.copy_(F.gelu(x[..., :d]) * x[..., d:])
    # Activation function without multiplication
    elif activation == "silu" and not is_gated:
        intermediate_cache2 = F.silu(intermediate_cache1.view(-1, N))
    elif activation == "gelu" and not is_gated:
        intermediate_cache2 = F.gelu(intermediate_cache1.view(-1, N))
    elif activation == "relu2" and not is_gated:
        intermediate_cache2 = torch.square(F.relu(intermediate_cache1.view(-1, N)))
    else:
        raise ValueError(f"Unsupported activation: {activation=}, with {is_gated=}")

    del intermediate_cache1

    intermediate_cache3 = torch.empty(
        (num_tokens, topk, w2.shape[1]),
        device=hidden_states.device,
        dtype=hidden_states.dtype,
    )

    # LoRA hooks force the second kernel to write to intermediate_cache3 so
    # hooks.after_down can inspect/modify it before reduction.
    _use_intermediate = not no_combine and (topk != 1 or hooks)

    out_slice = None
    if use_fused_moe_sum_all_reduce:
        out_slice = out_hidden_states
        out_slice.zero_()

View on GitHub (pinned to 0132848349)

Solutions

1. Use one of the supported combos: silu+gated, gelu+ungated, relu2+ungated
2. If the model needs gated gelu, extend the branch in fused_moe.py:805 area (add `elif activation == "gelu" and is_gated: ...` computing gate/up product) and file an upstream PR
3. Check the model config's hidden_act / act_fn for typos or unmapped names and map it to a supported one

Example fix

# before
fused_experts(..., activation="gelu", is_gated=True)  # raises

# after
fused_experts(..., activation="silu", is_gated=True)  # or add a gated-gelu branch

SUPPORTED_ACTIVATIONS = {("silu", True), ("gelu", False), ("relu2", False)}
assert (activation, bool(is_gated)) in SUPPORTED_ACTIVATIONS, \
    f"unsupported activation {activation} gated={is_gated}"

def is_supported_activation(name: str, gated: bool) -> bool:
    return (name, gated) in {("silu", True), ("gelu", False), ("relu2", False)}

When it happens

Trigger: Calling fused_experts_impl / the Triton runner's eager fallback with activation="gelu_tanh", "gelu" while is_gated=True, or any custom activation string; small-token debugging paths that force the torch branch; models registering a novel activation for their MoE experts.

Common situations: Adding a new MoE model whose experts use an activation SGLang hasn't mapped (e.g. gated GELU variants), or a typo in an activation name in a config; running with the fallback kernel path on debug runs.

Related errors

• num_token_non_padded must be a single-element tensor, got sh
• num_token_non_padded must be an integer tensor, got {num_tok
• num_token_non_padded and x must be on the same device
• Unsupported activation: {ACTIVATION_TYPE}
• topk_ids must be a CUDA tensor

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6e5947ee90ad9d9e. Report an issue: GitHub.