sgl-project/sglang · error · ValueError

Unsupported activation: {ACTIVATION_TYPE}

Error message

Unsupported activation: {ACTIVATION_TYPE}

What it means

ValueError raised at runtime inside the Triton-compiled _apply_activation function. The activation is baked in as the compile-time constant ACTIVATION_TYPE; only 'silu' and 'gelu' branches exist, and anything else hits the else branch raising this error.

Source

Thrown at python/sglang/kernels/ops/moe/fused_moe_triton_kernels.py:1079

def _apply_activation(x, ACTIVATION_TYPE: tl.constexpr):
    """
    Apply activation function based on compile-time constant.

    Args:
        x: Input tensor (converted to float32 inside)
        ACTIVATION_TYPE: Compile-time constant string ("silu" or "gelu")

    Returns:
        Activated output in the same dtype as input
    """
    x = x.to(tl.float32)
    if ACTIVATION_TYPE == "silu":
        return x * tl.sigmoid(x)
    elif ACTIVATION_TYPE == "gelu":
        kAlpha = 0.7978845608028654
        return 0.5 * x * (1 + tanh(kAlpha * (x + 0.044715 * x * x * x)))
    else:
        raise ValueError(f"Unsupported activation: {ACTIVATION_TYPE}")


@triton.jit
def act_and_mul_kernel(
    gateup_output,
    down_input,
    hidden_size,
    expert_ids_ptr,
    expert_step: tl.constexpr,
    BLOCK_SIZE: tl.constexpr,
    ACTIVATION_TYPE: tl.constexpr,
    SWIGLU_LIMIT: tl.constexpr = 0.0,
    HAS_SWIGLU_LIMIT: tl.constexpr = False,
    HAS_EXPERT_FILTER: tl.constexpr = True,
):
    """
    Unified activation and multiply kernel that handles both sorted and unsorted routing,
    and both SiLU and GELU activations using compile-time constants.

View on GitHub (pinned to 0132848349)

Solutions

1. Fix the activation string to exactly "silu" or "gelu" if one of those was intended
2. If a new activation is genuinely required, add an elif branch to _apply_activation in fused_moe_triton_kernels.py
3. Map nonstandard config names (e.g. gelu_new) to "gelu" at the caller before invoking the kernel

Example fix

// before
act_and_mul_kernel[grid](out, inp, "gelu_tanh", N, ...)
// after
# gelu_tanh ≈ the tanh approximation already implemented as "gelu"
act_and_mul_kernel[grid](out, inp, "gelu", N, ...)

assert activation in ("silu", "gelu"), f"unsupported activation {activation}"

def is_supported_activation(name: str) -> bool:
    return name in ("silu", "gelu")

try:
    act_and_mul_kernel[grid](out, inp, act, N)
except ValueError as e:
    if "Unsupported activation" in str(e):
        raise ValueError(f"model config requests {act!r}; kernel supports silu/gelu") from e
    raise

When it happens

Trigger: Passing an activation string other than "silu" or "gelu" (e.g. "gelu_tanh", "relu", "swiglu") to act_and_mul_kernel, which forwards it to _apply_activation as a constexpr.

Common situations: Adding a new gated-MLP activation to a model without extending this kernel; case/typo mistakes ("SiLU", "gelu-and"); a model config carrying an activation name the kernel never implemented.

Related errors

• num_token_non_padded must be a torch.Tensor
• num_token_non_padded must be a single-element tensor, got sh
• num_token_non_padded must be an integer tensor, got {num_tok
• topk kernels only support k <= 32: {k=}
• topk_ids must be int32, got {topk_ids.dtype}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f08451fd1d531a12. Report an issue: GitHub.