lllyasviel/Fooocus · error · RuntimeError

activation should be relu/gelu, not {activation}.

Error message

activation should be relu/gelu, not {activation}.

What it means

_get_activation_fn resolves the activation for CodeFormer's TransformerSALayer encoder layers. Only 'relu', 'gelu' and 'glu' are accepted; anything else raises RuntimeError. (The message mentions only relu/gelu, but the code also accepts 'glu' - the message is slightly stale.)

Source

Thrown at ldm_patched/pfn/architecture/face/codeformer.py:489

        pos_x = torch.stack(
            (pos_x[:, :, :, 0::2].sin(), pos_x[:, :, :, 1::2].cos()), dim=4
        ).flatten(3)
        pos_y = torch.stack(
            (pos_y[:, :, :, 0::2].sin(), pos_y[:, :, :, 1::2].cos()), dim=4
        ).flatten(3)
        pos = torch.cat((pos_y, pos_x), dim=3).permute(0, 3, 1, 2)
        return pos


def _get_activation_fn(activation):
    """Return an activation function given a string"""
    if activation == "relu":
        return F.relu
    if activation == "gelu":
        return F.gelu
    if activation == "glu":
        return F.glu
    raise RuntimeError(f"activation should be relu/gelu, not {activation}.")


class TransformerSALayer(nn.Module):
    def __init__(
        self, embed_dim, nhead=8, dim_mlp=2048, dropout=0.0, activation="gelu"
    ):
        super().__init__()
        self.self_attn = nn.MultiheadAttention(embed_dim, nhead, dropout=dropout)
        # Implementation of Feedforward model - MLP
        self.linear1 = nn.Linear(embed_dim, dim_mlp)
        self.dropout = nn.Dropout(dropout)
        self.linear2 = nn.Linear(dim_mlp, embed_dim)

        self.norm1 = nn.LayerNorm(embed_dim)
        self.norm2 = nn.LayerNorm(embed_dim)
        self.dropout1 = nn.Dropout(dropout)
        self.dropout2 = nn.Dropout(dropout)

View on GitHub (pinned to ae05379cc9)

Solutions

1. Use 'relu', 'gelu' or 'glu' exactly (lowercase)
2. If a different activation is required, add a branch returning the corresponding torch.nn.functional fn

Example fix

# before
layer = TransformerSALayer(embed_dim=256, activation='silu')
# -> RuntimeError: activation should be relu/gelu, not silu.

# after
layer = TransformerSALayer(embed_dim=256, activation='gelu')

SUPPORTED_ACTS = ('relu', 'gelu', 'glu')

activation = cfg.get('activation', 'gelu')
assert activation in SUPPORTED_ACTS, f'activation must be one of {SUPPORTED_ACTS} (case-sensitive, lowercase), got {activation!r}'

def is_supported_transformer_activation(name: str) -> bool:
    # case-sensitive: only exact lowercase relu/gelu/glu pass
    return name in ('relu', 'gelu', 'glu')

try:
    layer = TransformerSALayer(embed_dim=256, activation=cfg['activation'])
except RuntimeError as e:
    if 'activation should be relu/gelu' in str(e):
        cfg['activation'] = 'gelu'  # safe fallback for CodeFormer
        layer = TransformerSALayer(embed_dim=256, activation=cfg['activation'])
    else:
        raise

When it happens

Trigger: Constructing TransformerSALayer(..., activation='swish') or passing a config-derived activation string not in {'relu','gelu','glu'}; also fires on case-sensitive typos since, unlike the BasicSR factories, this function does NOT lowercase its input ('GELU' fails too).

Common situations: Porting DETR configs that add newer activations (selu, silu); configs using capitalized names; programmatic defaults like activation=None.

Related errors

• activation layer [{:s}] is not found
• normalize should be True if scale is passed
• checkpoint url or path is invalid
• The hidden size (%d) is not a multiple of the number of atte
• The hidden size (%d) is not a multiple of the number of atte

AI-assisted analysis of lllyasviel/Fooocus@ae05379cc9 (2026-08-15). Data as JSON: /api/errors/368d537a2a881dca. Report an issue: GitHub.