{"record":{"id":"368d537a2a881dca","repo":"lllyasviel/Fooocus","slug":"activation-should-be-relu-gelu-not-activation","errorCode":null,"errorMessage":"activation should be relu/gelu, not {activation}.","messagePattern":"activation should be relu/gelu, not (.+?)\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"ldm_patched/pfn/architecture/face/codeformer.py","lineNumber":489,"sourceCode":"        pos_x = torch.stack(\n            (pos_x[:, :, :, 0::2].sin(), pos_x[:, :, :, 1::2].cos()), dim=4\n        ).flatten(3)\n        pos_y = torch.stack(\n            (pos_y[:, :, :, 0::2].sin(), pos_y[:, :, :, 1::2].cos()), dim=4\n        ).flatten(3)\n        pos = torch.cat((pos_y, pos_x), dim=3).permute(0, 3, 1, 2)\n        return pos\n\n\ndef _get_activation_fn(activation):\n    \"\"\"Return an activation function given a string\"\"\"\n    if activation == \"relu\":\n        return F.relu\n    if activation == \"gelu\":\n        return F.gelu\n    if activation == \"glu\":\n        return F.glu\n    raise RuntimeError(f\"activation should be relu/gelu, not {activation}.\")\n\n\nclass TransformerSALayer(nn.Module):\n    def __init__(\n        self, embed_dim, nhead=8, dim_mlp=2048, dropout=0.0, activation=\"gelu\"\n    ):\n        super().__init__()\n        self.self_attn = nn.MultiheadAttention(embed_dim, nhead, dropout=dropout)\n        # Implementation of Feedforward model - MLP\n        self.linear1 = nn.Linear(embed_dim, dim_mlp)\n        self.dropout = nn.Dropout(dropout)\n        self.linear2 = nn.Linear(dim_mlp, embed_dim)\n\n        self.norm1 = nn.LayerNorm(embed_dim)\n        self.norm2 = nn.LayerNorm(embed_dim)\n        self.dropout1 = nn.Dropout(dropout)\n        self.dropout2 = nn.Dropout(dropout)\n","sourceCodeStart":471,"sourceCodeEnd":507,"githubUrl":"https://github.com/lllyasviel/Fooocus/blob/ae05379cc97bc4361ec8b4ec90193dab21be763f/ldm_patched/pfn/architecture/face/codeformer.py#L471-L507","documentation":"_get_activation_fn resolves the activation for CodeFormer's TransformerSALayer encoder layers. Only 'relu', 'gelu' and 'glu' are accepted; anything else raises RuntimeError. (The message mentions only relu/gelu, but the code also accepts 'glu' - the message is slightly stale.)","triggerScenarios":"Constructing TransformerSALayer(..., activation='swish') or passing a config-derived activation string not in {'relu','gelu','glu'}; also fires on case-sensitive typos since, unlike the BasicSR factories, this function does NOT lowercase its input ('GELU' fails too).","commonSituations":"Porting DETR configs that add newer activations (selu, silu); configs using capitalized names; programmatic defaults like activation=None.","solutions":["Use 'relu', 'gelu' or 'glu' exactly (lowercase)","If a different activation is required, add a branch returning the corresponding torch.nn.functional fn"],"exampleFix":"# before\nlayer = TransformerSALayer(embed_dim=256, activation='silu')\n# -> RuntimeError: activation should be relu/gelu, not silu.\n\n# after\nlayer = TransformerSALayer(embed_dim=256, activation='gelu')","handlingStrategy":"validation","validationCode":"SUPPORTED_ACTS = ('relu', 'gelu', 'glu')\n\nactivation = cfg.get('activation', 'gelu')\nassert activation in SUPPORTED_ACTS, f'activation must be one of {SUPPORTED_ACTS} (case-sensitive, lowercase), got {activation!r}'","typeGuard":"def is_supported_transformer_activation(name: str) -> bool:\n    # case-sensitive: only exact lowercase relu/gelu/glu pass\n    return name in ('relu', 'gelu', 'glu')","tryCatchPattern":"try:\n    layer = TransformerSALayer(embed_dim=256, activation=cfg['activation'])\nexcept RuntimeError as e:\n    if 'activation should be relu/gelu' in str(e):\n        cfg['activation'] = 'gelu'  # safe fallback for CodeFormer\n        layer = TransformerSALayer(embed_dim=256, activation=cfg['activation'])\n    else:\n        raise","preventionTips":["Pass activation exactly as lowercase 'relu', 'gelu' or 'glu' (matching is case-sensitive here)","Whitelist the activation string in your config loader; default to 'gelu' for CodeFormer"],"tags":["codeformer","activation","transformer","config","case-sensitive"],"backgroundTag":null,"analyzedSha":"ae05379cc97bc4361ec8b4ec90193dab21be763f","analyzedAt":"2026-08-15T04:23:59.533Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}