What causes "Unsupported activation function: {}" in huggingface/candle?

Set activation_function to "gelu" or "relu" in VoxtralEncoderConfig

Unsupported activation function: {}

Error message

Unsupported activation function: {}

What it means

The Voxtral encoder feed-forward block maps the config's activation_function string to candle_nn::Activation, supporting only "gelu" and "relu". Any other value bails with the unsupported string. Candle intentionally whitelists activations rather than attempting to parse arbitrary names.

Source

Thrown at candle-transformers/src/models/voxtral/model.rs:376

    activation: candle_nn::Activation,
    dropout: Dropout,
    activation_dropout: Dropout,
}

impl VoxtralEncoderLayer {
    fn new(cfg: &VoxtralEncoderConfig, vb: VarBuilder) -> Result<Self> {
        let embed_dim = cfg.hidden_size;

        let self_attn = VoxtralAttention::new(cfg, vb.pp("self_attn"))?;
        let self_attn_layer_norm = layer_norm(embed_dim, 1e-5, vb.pp("self_attn_layer_norm"))?;
        let fc1 = linear(embed_dim, cfg.intermediate_size, vb.pp("fc1"))?;
        let fc2 = linear(cfg.intermediate_size, embed_dim, vb.pp("fc2"))?;
        let final_layer_norm = layer_norm(embed_dim, 1e-5, vb.pp("final_layer_norm"))?;

        let activation = match cfg.activation_function.as_str() {
            "gelu" => candle_nn::Activation::Gelu,
            "relu" => candle_nn::Activation::Relu,
            _ => candle::bail!(
                "Unsupported activation function: {}",
                cfg.activation_function
            ),
        };

        let dropout = Dropout::new(cfg.dropout as f32);
        let activation_dropout = Dropout::new(cfg.activation_dropout as f32);

        Ok(Self {
            self_attn,
            self_attn_layer_norm,
            fc1,
            fc2,
            final_layer_norm,
            activation,
            dropout,
            activation_dropout,
        })

View on GitHub (pinned to d5fee525bf)

Solutions

Set activation_function to "gelu" or "relu" in VoxtralEncoderConfig
If the checkpoint uses e.g. "silu", patch the model code to map it to candle_nn::Activation::Silu
Confirm the checkpoint's actual activation in its config.json and adjust

Example fix

// before
let cfg = VoxtralEncoderConfig { activation_function: "gelu_new".into(), .. };
// after
let cfg = VoxtralEncoderConfig { activation_function: "gelu".into(), .. };

Defensive patterns

Strategy: validation

Validate before calling

if !matches!(cfg.activation_function.as_str(), "gelu" | "relu") {
    return Err(anyhow::anyhow!("activation {:?} unsupported by candle Voxtral encoder", cfg.activation_function));
}

Try / catch

match VoxtralEncoder::new(&cfg, vb) {
    Err(e) if e.to_string().contains("Unsupported activation function") => {
        anyhow::bail!("map {:?} to gelu/relu or patch the model", cfg.activation_function)
    }
    r => r?,
}

Prevention

Whitelist activations in your config-loading layer before model construction
Map common HF aliases (gelu_new -> gelu, silu -> Silu) when porting configs
Check config.json of the checkpoint for its declared activation

When it happens

Trigger: Constructing the Voxtral encoder FFN with a VoxtralEncoderConfig whose activation_function is a string other than "gelu" or "relu" (e.g. "gelu_new", "silu", "swiglu").

Common situations: Using a checkpoint whose HF config declares gelu_new or silu; copying config values from another model family; typos in a hand-written config.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/1b8de1a95a7348b2. Report an issue: GitHub.