{"record":{"id":"f0b2cc853f536a7c","repo":"sgl-project/sglang","slug":"unsupported-activation-hidden-act-only-silu-is-f0b2cc","errorCode":null,"errorMessage":"Unsupported activation: {hidden_act}. Only silu is supported for now.","messagePattern":"Unsupported activation: (.+?)\\. Only silu is supported for now\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/torch_native_llama.py","lineNumber":128,"sourceCode":"        intermediate_size: int,\n        hidden_act: str,\n        quant_config: Optional[QuantizationConfig] = None,\n        prefix: str = \"\",\n    ) -> None:\n        super().__init__()\n        self.gate_up_proj = torch.nn.Linear(\n            hidden_size,\n            intermediate_size * 2,\n            bias=False,\n        )\n        self.gate_up_proj.output_sizes = [intermediate_size] * 2\n        self.gate_up_proj.weight_loader = types.MethodType(\n            gate_up_proj_weight_loader, self.gate_up_proj\n        )\n        self.gate_up_proj.weight.weight_loader = self.gate_up_proj.weight_loader\n        self.down_proj = torch.nn.Linear(intermediate_size, hidden_size, bias=False)\n        if hidden_act != \"silu\":\n            raise ValueError(\n                f\"Unsupported activation: {hidden_act}. \"\n                \"Only silu is supported for now.\"\n            )\n        self.act_fn = SiluAndMul()\n\n    def forward(self, x):\n        gate_up = self.gate_up_proj(x)\n        x = self.act_fn(gate_up)\n        x = self.down_proj(x)\n        return x\n\n\ndef qkv_proj_weight_loader(\n    self,\n    param: Parameter,\n    loaded_weight: torch.Tensor,\n    loaded_shard_id: str,\n):","sourceCodeStart":110,"sourceCodeEnd":146,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/torch_native_llama.py#L110-L146","documentation":"The torch-native Llama MLP only implements SiluAndMul; any hidden_activation other than 'silu' in the config is rejected at construction time.","triggerScenarios":"Instantiating TorchNativeLlamaMLP with config.hidden_activation in {'gelu','gelu_pytorch_tanh','relu',...}.","commonSituations":"Pointing the native-torch fallback path at non-LLaMA/silu models; newer HF configs defaulting to gelu variants.","solutions":["Use the regular sglang model implementation for that architecture instead of torch_native_llama","Set hidden_activation='silu' in the config if the model truly uses SwiGLU","Extend the class to support the needed activation"],"exampleFix":"// before\nself.act_fn = SiluAndMul()  # after raise for non-silu\n// after\nACT = {\"silu\": SiluAndMul}\nself.act_fn = ACT[hidden_act]()  # after adding mappings","handlingStrategy":"validation","validationCode":"assert config.hidden_activation == 'silu', 'torch_native_llama only supports silu'","typeGuard":"def supports_native_llama(cfg) -> bool:\n    return getattr(cfg, 'hidden_activation', 'silu') == 'silu'","tryCatchPattern":null,"preventionTips":["Check hidden_activation before selecting native path","Default to the architecture-specific sglang model"],"tags":["activation-function","config-validation","llama"],"backgroundTag":"unsupported-activation-function","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}