sgl-project/sglang · error · ValueError

Unsupported activation: {hidden_act}. Only silu is supported

Error message

Unsupported activation: {hidden_act}. Only silu is supported for now.

What it means

InternLM2 MLP rejects any hidden_act other than 'silu' because its forward path hard-codes SiluAndMul. The check runs in __init__ right before constructing the activation.

Source

Thrown at python/sglang/srt/models/internlm2.py:72

        prefix: str = "",
    ) -> None:
        super().__init__()
        self.gate_up_proj = MergedColumnParallelLinear(
            hidden_size,
            [intermediate_size] * 2,
            bias=False,
            quant_config=quant_config,
            prefix=add_prefix("gate_up_proj", prefix),
        )
        self.w2 = RowParallelLinear(
            intermediate_size,
            hidden_size,
            bias=False,
            quant_config=quant_config,
            prefix=add_prefix("w2", prefix),
        )
        if hidden_act != "silu":
            raise ValueError(
                f"Unsupported activation: {hidden_act}. "
                "Only silu is supported for now."
            )
        self.act_fn = SiluAndMul()

    def forward(self, x):
        gate_up, _ = self.gate_up_proj(x)
        x = self.act_fn(gate_up)
        x, _ = self.w2(x)
        return x


class InternLM2Attention(nn.Module):
    def __init__(
        self,
        hidden_size: int,
        num_heads: int,
        num_kv_heads: int,

View on GitHub (pinned to 0132848349)

Solutions

1. Set hidden_act='silu' in the model config (correct for stock InternLM2)
2. If the checkpoint truly uses another activation, patch the model class to support it before using it

Example fix

# before
config.hidden_act = "gelu"
# after
config.hidden_act = "silu"

assert config.hidden_act == 'silu', config.hidden_act

When it happens

Trigger: Loading an InternLM2 config whose hidden_act is e.g. 'gelu', 'gelu_new', or misspelled.

Common situations: Custom/finetuned InternLM2 checkpoint edited to a different activation; config typo in hidden_act.

Related errors

• Unsupported activation: {activation_type}
• Checkpoint provides gate weight {name!r} but the model built
• Unsupported activation: {ACTIVATION_TYPE}
• native MXFP8 MoE only supports gated swiglu-oai, got {activa
• expected a tensor with at least one dimension

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9531026f1a9c00a4. Report an issue: GitHub.