labmlai/annotated_deep_learning_paper_implementations · error · ValueError

Unknown variant {configs.glu_variant}

Error message

Unknown variant {configs.glu_variant}

What it means

The GLU-variants transformer experiment builds its FFN from configs.glu_variant via a chain of string comparisons (e.g. 'Bilinear', 'ReGLU', 'GeGLU', 'SwiGLU', plain 'ReLU'/'GELU'). Any string not matching a known branch falls through to the else and raises ValueError with the offending value. It is a config-enum validation error: the variant name is misspelled, wrongly cased, or not implemented in this experiment.

Source

Thrown at labml_nn/transformers/glu_variants/simple.py:173

        # FFN with GELU gate
        # $$FFN_{GEGLU}(x)(x, W_1, V, W_2) = (\text{GELU}(x W_1) \otimes x V) W_2$$
        elif configs.glu_variant == 'GEGLU':
            ffn = FeedForward(configs.d_model, configs.d_ff, configs.dropout, nn.GELU(), True, False, False, False)
        # FFN with Swish gate
        # $$FFN_{SwiGLU}(x)(x, W_1, V, W_2) = (\text{Swish}_1(x W_1) \otimes x V) W_2$$
        # where $\text{Swish}_\beta(x) = x \sigma(\beta x)$
        elif configs.glu_variant == 'SwiGLU':
            ffn = FeedForward(configs.d_model, configs.d_ff, configs.dropout, nn.SiLU(), True, False, False, False)
        # FFN with ReLU activation
        # $$FFN_{ReLU}(x)(x, W_1, W_2, b_1, b_2) = \text{ReLU}_1(x W_1 + b_1) W_2 + b_2$$
        elif configs.glu_variant == 'ReLU':
            ffn = FeedForward(configs.d_model, configs.d_ff, configs.dropout, nn.ReLU())
        # FFN with ReLU activation
        # $$FFN_{GELU}(x)(x, W_1, W_2, b_1, b_2) = \text{GELU}_1(x W_1 + b_1) W_2 + b_2$$
        elif configs.glu_variant == 'GELU':
            ffn = FeedForward(configs.d_model, configs.d_ff, configs.dropout, nn.GELU())
        else:
            raise ValueError(f'Unknown variant {configs.glu_variant}')

        # Number of different characters
        n_chars = len(self.dataset.stoi)

        # Initialize [Multi-Head Attention module](../mha.html)
        mha = MultiHeadAttention(configs.n_heads, configs.d_model, configs.dropout)
        # Initialize the [Transformer Block](../models.html#TransformerLayer)
        transformer_layer = TransformerLayer(d_model=configs.d_model, self_attn=mha, src_attn=None,
                                             feed_forward=ffn, dropout_prob=configs.dropout)
        # Initialize the model with an
        # [embedding layer](../models.html#EmbeddingsWithPositionalEncoding)
        # (with fixed positional encoding)
        # [transformer encoder](../models.html#Encoder) and
        # a linear layer to generate logits.
        self.model = AutoregressiveModel(EmbeddingsWithPositionalEncoding(configs.d_model, n_chars),
                                         Encoder(transformer_layer, configs.n_layers),
                                         nn.Linear(configs.d_model, n_chars))

View on GitHub (pinned to 33ab02281c)

Solutions

1. Set glu_variant to one of the supported exact strings: None, 'Bilinear', 'ReGLU', 'GeGLU', 'SwiGLU', 'ReLU', 'GELU'
2. Check for casing/whitespace typos in the config value (comparison is case-sensitive)
3. If you need a custom activation, extend the if/elif chain in simple.py with your own branch

Example fix

# before: raises Unknown variant
configs.glu_variant = 'reglu'

# after
configs.glu_variant = 'ReGLU'

SUPPORTED = {None, 'Bilinear', 'ReLU', 'GELU', 'ReGLU', 'GeGLU', 'SwiGLU'}
variant = configs.glu_variant
if variant not in SUPPORTED:
    raise ValueError(f'glu_variant must be one of {sorted(map(str, SUPPORTED))}, got {variant!r}')

def is_supported_glu_variant(name):
    return name in {None, 'Bilinear', 'ReLU', 'GELU', 'ReGLU', 'GeGLU', 'SwiGLU'}

try:
    experiment = Configs()
    labml.experiment.run()
except ValueError as e:
    if 'Unknown variant' in str(e):
        raise SystemExit(f'Fix configs.glu_variant: {e}')
    raise

When it happens

Trigger: Running the glu_variants experiment with configs.glu_variant set to an unsupported or misspelled string, e.g. 'glu', 'Reglu', 'geglu ', 'swish-glu', or a variant added in another repo but not here.

Common situations: Passing the variant via labml experiment CLI/config file with different casing or a trailing space; porting a variant name from the GLU paper or another codebase; expecting a newly published variant (e.g. 'xSwiGLU') that this experiment never implemented.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• Invalid learning rate: {lr}
• Invalid epsilon value: {eps}
• Invalid beta parameter at index 0: {betas[0]}
• Invalid beta parameter at index 1: {betas[1]}
• Invalid weight_decay value: {weight_decay}

AI-assisted analysis of labmlai/annotated_deep_learning_paper_implementations@33ab02281c (2026-08-25). Data as JSON: /api/errors/8a20273b4766a1c3. Report an issue: GitHub.