{"record":{"id":"6b3742aabc76ba86","repo":"dotnet/machinelearning","slug":"invalid-activation-function","errorCode":null,"errorMessage":"Invalid activation function","messagePattern":"Invalid activation function","errorType":"validation","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.GenAI.Core/Utils.cs","lineNumber":129,"sourceCode":"        var qEmbed = q * cos;\n        qEmbed += RotateHalf(q) * sin;\n\n        var kEmbed = k * cos;\n        kEmbed += RotateHalf(k) * sin;\n        // var kEmbed = (k * cos) + (RotateHalf(k) * sin);\n        return (qEmbed, kEmbed);\n    }\n\n    public static Module<Tensor, Tensor> GetActivation(string actFn)\n    {\n        return actFn switch\n        {\n            \"silu\" => nn.SiLU(),\n            \"relu\" => nn.ReLU(),\n            \"gelu\" => nn.GELU(),\n            \"tanh\" => nn.Tanh(),\n            \"swish\" => nn.SiLU(),\n            _ => throw new ArgumentException(\"Invalid activation function\", actFn),\n        };\n    }\n\n\n    public static Tensor Phi2RepeatKV(Tensor x, int nRep)\n    {\n        var batchSize = x.shape[0];\n        var seqLen = x.shape[1];\n        var nKVHeads = x.shape[2];\n        var headDim = x.shape[3];\n        if (nRep == 1)\n        {\n            return x;\n        }\n\n        return x.unsqueeze(3)\n                .expand(batchSize, seqLen, nKVHeads, nRep, headDim)\n                .view(batchSize, seqLen, nKVHeads * nRep, headDim);","sourceCodeStart":111,"sourceCodeEnd":147,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.GenAI.Core/Utils.cs#L111-L147","documentation":"Utils.GetActivation maps a string activation name to a TorchSharp nn module and throws ArgumentException for unrecognized names. Only silu, relu, gelu, tanh, and swish (aliased to SiLU) are supported; anything else — including common variants like 'gelu_new', 'relu2', or capitalized names — is rejected.","triggerScenarios":"Loading a model config whose hidden_activation / act_fn value is not one of the five supported strings, e.g. 'GELU', 'gelu_new', 'quick_gelu', or 'sigmoid'.","commonSituations":"Porting checkpoints from architectures (Phi, Gemma, Qwen) that use activations like gelu_new or quick_gelu not present in the switch; case-sensitivity issues when copying activation names from HF configs.","solutions":["Change the config's activation value to one of the supported names: silu, relu, gelu, tanh, or swish.","Normalize casing before calling (e.g. actFn.ToLowerInvariant()) so 'GELU' works.","For unsupported variants like gelu_new or quick_gelu, extend GetActivation with the corresponding nn module (e.g. approximate GELU).","Check the HF config.json for the exact activation name and map it manually to the closest supported one."],"exampleFix":"// before\nstring actFn = \"gelu_new\";\nvar act = Utils.GetActivation(actFn); // throws\n// after\nstring actFn = \"gelu_new\".ToLowerInvariant();\nvar act = actFn == \"gelu_new\" ? nn.GELU(approximate: GELUApproximationMode.Tanh) : Utils.GetActivation(actFn);","handlingStrategy":"validation","validationCode":"string[] supported = { \"silu\", \"relu\", \"gelu\", \"tanh\", \"swish\" };\nif (!supported.Contains(actFn?.ToLowerInvariant())) throw new NotSupportedException($\"activation '{actFn}' not supported; use one of: {string.Join(\",\", supported)}\");","typeGuard":"bool IsSupportedActivation(string? actFn) => actFn is not null && new[]{\"silu\",\"relu\",\"gelu\",\"tanh\",\"swish\"}.Contains(actFn.ToLowerInvariant());","tryCatchPattern":"try { var act = Utils.GetActivation(actFn); }\ncatch (ArgumentException ex) when (ex.Message.Contains(\"Invalid activation function\")) { logger.LogError(\"Unsupported activation '{ActFn}' from config; map it manually\", actFn); throw; }","preventionTips":["Normalize activation names to lowercase before passing.","Map HF config names (gelu_new, quick_gelu) to supported equivalents when porting checkpoints.","Keep a lookup table of unsupported architectures' activations.","Add missing nn modules to GetActivation for architectures you target."],"tags":["activation","config","argument"],"backgroundTag":"invalid-enum-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}