dotnet/machinelearning · error · ArgumentException

Invalid activation function

Error message

Invalid activation function

What it means

Utils.GetActivation maps a string activation name to a TorchSharp nn module and throws ArgumentException for unrecognized names. Only silu, relu, gelu, tanh, and swish (aliased to SiLU) are supported; anything else — including common variants like 'gelu_new', 'relu2', or capitalized names — is rejected.

Solutions

1. Change the config's activation value to one of the supported names: silu, relu, gelu, tanh, or swish.
2. Normalize casing before calling (e.g. actFn.ToLowerInvariant()) so 'GELU' works.
3. For unsupported variants like gelu_new or quick_gelu, extend GetActivation with the corresponding nn module (e.g. approximate GELU).
4. Check the HF config.json for the exact activation name and map it manually to the closest supported one.

Example fix

// before
string actFn = "gelu_new";
var act = Utils.GetActivation(actFn); // throws
// after
string actFn = "gelu_new".ToLowerInvariant();
var act = actFn == "gelu_new" ? nn.GELU(approximate: GELUApproximationMode.Tanh) : Utils.GetActivation(actFn);

string[] supported = { "silu", "relu", "gelu", "tanh", "swish" };
if (!supported.Contains(actFn?.ToLowerInvariant())) throw new NotSupportedException($"activation '{actFn}' not supported; use one of: {string.Join(",", supported)}");

bool IsSupportedActivation(string? actFn) => actFn is not null && new[]{"silu","relu","gelu","tanh","swish"}.Contains(actFn.ToLowerInvariant());

try { var act = Utils.GetActivation(actFn); }
catch (ArgumentException ex) when (ex.Message.Contains("Invalid activation function")) { logger.LogError("Unsupported activation '{ActFn}' from config; map it manually", actFn); throw; }

When it happens

Trigger: Loading a model config whose hidden_activation / act_fn value is not one of the five supported strings, e.g. 'GELU', 'gelu_new', 'quick_gelu', or 'sigmoid'.

Common situations: Porting checkpoints from architectures (Phi, Gemma, Qwen) that use activations like gelu_new or quick_gelu not present in the switch; case-sensitivity issues when copying activation names from HF configs.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

• Activation function not supported.
• Array lengths are mistmached
• Blob for normalization rule is broken.
• Column lengths are mismatched
• Column lengths are mismatched

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/6b3742aabc76ba86. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.GenAI.Core/Utils.cs:129

        var qEmbed = q * cos;
        qEmbed += RotateHalf(q) * sin;

        var kEmbed = k * cos;
        kEmbed += RotateHalf(k) * sin;
        // var kEmbed = (k * cos) + (RotateHalf(k) * sin);
        return (qEmbed, kEmbed);
    }

    public static Module<Tensor, Tensor> GetActivation(string actFn)
    {
        return actFn switch
        {
            "silu" => nn.SiLU(),
            "relu" => nn.ReLU(),
            "gelu" => nn.GELU(),
            "tanh" => nn.Tanh(),
            "swish" => nn.SiLU(),
            _ => throw new ArgumentException("Invalid activation function", actFn),
        };
    }


    public static Tensor Phi2RepeatKV(Tensor x, int nRep)
    {
        var batchSize = x.shape[0];
        var seqLen = x.shape[1];
        var nKVHeads = x.shape[2];
        var headDim = x.shape[3];
        if (nRep == 1)
        {
            return x;
        }

        return x.unsqueeze(3)
                .expand(batchSize, seqLen, nKVHeads, nRep, headDim)
                .view(batchSize, seqLen, nKVHeads * nRep, headDim);

View on GitHub (pinned to 7b76e69cf9)