sgl-project/sglang · error · ValueError

Unsupported activation: {hidden_act}. Only silu is supported

Error message

Unsupported activation: {hidden_act}. Only silu is supported for now.

What it means

The torch-native Llama MLP only implements SiluAndMul; any hidden_activation other than 'silu' in the config is rejected at construction time.

Source

Thrown at python/sglang/srt/models/torch_native_llama.py:128

        intermediate_size: int,
        hidden_act: str,
        quant_config: Optional[QuantizationConfig] = None,
        prefix: str = "",
    ) -> None:
        super().__init__()
        self.gate_up_proj = torch.nn.Linear(
            hidden_size,
            intermediate_size * 2,
            bias=False,
        )
        self.gate_up_proj.output_sizes = [intermediate_size] * 2
        self.gate_up_proj.weight_loader = types.MethodType(
            gate_up_proj_weight_loader, self.gate_up_proj
        )
        self.gate_up_proj.weight.weight_loader = self.gate_up_proj.weight_loader
        self.down_proj = torch.nn.Linear(intermediate_size, hidden_size, bias=False)
        if hidden_act != "silu":
            raise ValueError(
                f"Unsupported activation: {hidden_act}. "
                "Only silu is supported for now."
            )
        self.act_fn = SiluAndMul()

    def forward(self, x):
        gate_up = self.gate_up_proj(x)
        x = self.act_fn(gate_up)
        x = self.down_proj(x)
        return x


def qkv_proj_weight_loader(
    self,
    param: Parameter,
    loaded_weight: torch.Tensor,
    loaded_shard_id: str,
):

View on GitHub (pinned to 0132848349)

Solutions

1. Use the regular sglang model implementation for that architecture instead of torch_native_llama
2. Set hidden_activation='silu' in the config if the model truly uses SwiGLU
3. Extend the class to support the needed activation

Example fix

// before
self.act_fn = SiluAndMul()  # after raise for non-silu
// after
ACT = {"silu": SiluAndMul}
self.act_fn = ACT[hidden_act]()  # after adding mappings

assert config.hidden_activation == 'silu', 'torch_native_llama only supports silu'

def supports_native_llama(cfg) -> bool:
    return getattr(cfg, 'hidden_activation', 'silu') == 'silu'

When it happens

Trigger: Instantiating TorchNativeLlamaMLP with config.hidden_activation in {'gelu','gelu_pytorch_tanh','relu',...}.

Common situations: Pointing the native-torch fallback path at non-LLaMA/silu models; newer HF configs defaulting to gelu variants.

Related errors

• bad compress_ratio {compress_ratio}
• The requested FlashAttention forward configuration exceeds S
• flashinfer_sparse_mla supports only GLM DSA with FP8 KV cach
• GLM DSA with FP8 KV cache on NVIDIA SM120/SM121 supports onl
• kv-canary: RealKvSource.page_size must be >= 1, got {self.pa

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f0b2cc853f536a7c. Report an issue: GitHub.