{"record":{"id":"8a20273b4766a1c3","repo":"labmlai/annotated_deep_learning_paper_implementations","slug":"unknown-variant-configs-glu-variant","errorCode":null,"errorMessage":"Unknown variant {configs.glu_variant}","messagePattern":"Unknown variant (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"labml_nn/transformers/glu_variants/simple.py","lineNumber":173,"sourceCode":"        # FFN with GELU gate\n        # $$FFN_{GEGLU}(x)(x, W_1, V, W_2) = (\\text{GELU}(x W_1) \\otimes x V) W_2$$\n        elif configs.glu_variant == 'GEGLU':\n            ffn = FeedForward(configs.d_model, configs.d_ff, configs.dropout, nn.GELU(), True, False, False, False)\n        # FFN with Swish gate\n        # $$FFN_{SwiGLU}(x)(x, W_1, V, W_2) = (\\text{Swish}_1(x W_1) \\otimes x V) W_2$$\n        # where $\\text{Swish}_\\beta(x) = x \\sigma(\\beta x)$\n        elif configs.glu_variant == 'SwiGLU':\n            ffn = FeedForward(configs.d_model, configs.d_ff, configs.dropout, nn.SiLU(), True, False, False, False)\n        # FFN with ReLU activation\n        # $$FFN_{ReLU}(x)(x, W_1, W_2, b_1, b_2) = \\text{ReLU}_1(x W_1 + b_1) W_2 + b_2$$\n        elif configs.glu_variant == 'ReLU':\n            ffn = FeedForward(configs.d_model, configs.d_ff, configs.dropout, nn.ReLU())\n        # FFN with ReLU activation\n        # $$FFN_{GELU}(x)(x, W_1, W_2, b_1, b_2) = \\text{GELU}_1(x W_1 + b_1) W_2 + b_2$$\n        elif configs.glu_variant == 'GELU':\n            ffn = FeedForward(configs.d_model, configs.d_ff, configs.dropout, nn.GELU())\n        else:\n            raise ValueError(f'Unknown variant {configs.glu_variant}')\n\n        # Number of different characters\n        n_chars = len(self.dataset.stoi)\n\n        # Initialize [Multi-Head Attention module](../mha.html)\n        mha = MultiHeadAttention(configs.n_heads, configs.d_model, configs.dropout)\n        # Initialize the [Transformer Block](../models.html#TransformerLayer)\n        transformer_layer = TransformerLayer(d_model=configs.d_model, self_attn=mha, src_attn=None,\n                                             feed_forward=ffn, dropout_prob=configs.dropout)\n        # Initialize the model with an\n        # [embedding layer](../models.html#EmbeddingsWithPositionalEncoding)\n        # (with fixed positional encoding)\n        # [transformer encoder](../models.html#Encoder) and\n        # a linear layer to generate logits.\n        self.model = AutoregressiveModel(EmbeddingsWithPositionalEncoding(configs.d_model, n_chars),\n                                         Encoder(transformer_layer, configs.n_layers),\n                                         nn.Linear(configs.d_model, n_chars))\n","sourceCodeStart":155,"sourceCodeEnd":191,"githubUrl":"https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/33ab02281c2b928e6b32792909cc79cbdcfe1d6a/labml_nn/transformers/glu_variants/simple.py#L155-L191","documentation":"The GLU-variants transformer experiment builds its FFN from configs.glu_variant via a chain of string comparisons (e.g. 'Bilinear', 'ReGLU', 'GeGLU', 'SwiGLU', plain 'ReLU'/'GELU'). Any string not matching a known branch falls through to the else and raises ValueError with the offending value. It is a config-enum validation error: the variant name is misspelled, wrongly cased, or not implemented in this experiment.","triggerScenarios":"Running the glu_variants experiment with configs.glu_variant set to an unsupported or misspelled string, e.g. 'glu', 'Reglu', 'geglu ', 'swish-glu', or a variant added in another repo but not here.","commonSituations":"Passing the variant via labml experiment CLI/config file with different casing or a trailing space; porting a variant name from the GLU paper or another codebase; expecting a newly published variant (e.g. 'xSwiGLU') that this experiment never implemented.","solutions":["Set glu_variant to one of the supported exact strings: None, 'Bilinear', 'ReGLU', 'GeGLU', 'SwiGLU', 'ReLU', 'GELU'","Check for casing/whitespace typos in the config value (comparison is case-sensitive)","If you need a custom activation, extend the if/elif chain in simple.py with your own branch"],"exampleFix":"# before: raises Unknown variant\nconfigs.glu_variant = 'reglu'\n\n# after\nconfigs.glu_variant = 'ReGLU'","handlingStrategy":"type-guard","validationCode":"SUPPORTED = {None, 'Bilinear', 'ReLU', 'GELU', 'ReGLU', 'GeGLU', 'SwiGLU'}\nvariant = configs.glu_variant\nif variant not in SUPPORTED:\n    raise ValueError(f'glu_variant must be one of {sorted(map(str, SUPPORTED))}, got {variant!r}')","typeGuard":"def is_supported_glu_variant(name):\n    return name in {None, 'Bilinear', 'ReLU', 'GELU', 'ReGLU', 'GeGLU', 'SwiGLU'}","tryCatchPattern":"try:\n    experiment = Configs()\n    labml.experiment.run()\nexcept ValueError as e:\n    if 'Unknown variant' in str(e):\n        raise SystemExit(f'Fix configs.glu_variant: {e}')\n    raise","preventionTips":["Copy variant names verbatim from the experiment source (case-sensitive)","Strip whitespace and normalize case when reading variant strings from CLI/config files","Keep the supported-variant set in one constant shared by config validation and model construction"],"tags":["python","pytorch","transformer","glu-variants","config","validation"],"backgroundTag":"invalid-config-value","analyzedSha":"33ab02281c2b928e6b32792909cc79cbdcfe1d6a","analyzedAt":"2026-08-25T10:30:27.743Z","schemaVersion":2},"datasetVersion":"2026-08-25T11:17:15.655Z"}