lllyasviel/Fooocus · error · ValueError

The hidden size (%d) is not a multiple of the number of atte

Error message

The hidden size (%d) is not a multiple of the number of attention heads (%d)

What it means

Same constructor validation as the MED BertSelfAttention but in the NLVR twin-encoder copy (nlvr_encoder.py): hidden_size must be divisible by num_attention_heads so each head receives an integer slice of the embedding for Q/K/V projections; otherwise model construction fails immediately.

Source

Thrown at extras/BLIP/models/nlvr_encoder.py:92

        if inputs_embeds is None:
            inputs_embeds = self.word_embeddings(input_ids)

        embeddings = inputs_embeds

        if self.position_embedding_type == "absolute":
            position_embeddings = self.position_embeddings(position_ids)
            embeddings += position_embeddings
        embeddings = self.LayerNorm(embeddings)
        embeddings = self.dropout(embeddings)
        return embeddings


class BertSelfAttention(nn.Module):
    def __init__(self, config, is_cross_attention):
        super().__init__()
        self.config = config
        if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, "embedding_size"):
            raise ValueError(
                "The hidden size (%d) is not a multiple of the number of attention "
                "heads (%d)" % (config.hidden_size, config.num_attention_heads)
            )
        
        self.num_attention_heads = config.num_attention_heads
        self.attention_head_size = int(config.hidden_size / config.num_attention_heads)
        self.all_head_size = self.num_attention_heads * self.attention_head_size

        self.query = nn.Linear(config.hidden_size, self.all_head_size)
        if is_cross_attention:
            self.key = nn.Linear(config.encoder_width, self.all_head_size)
            self.value = nn.Linear(config.encoder_width, self.all_head_size)
        else:
            self.key = nn.Linear(config.hidden_size, self.all_head_size)
            self.value = nn.Linear(config.hidden_size, self.all_head_size)

        self.dropout = nn.Dropout(config.attention_probs_dropout_prob)
        self.position_embedding_type = getattr(config, "position_embedding_type", "absolute")

View on GitHub (pinned to ae05379cc9)

Solutions

1. Fix the config: choose num_attention_heads that divides hidden_size
2. Recompute both values together when scaling the model
3. Validate config invariants before instantiating the model

Example fix

// before
cfg = BertConfig(hidden_size=1024, num_attention_heads=48)

// after
cfg = BertConfig(hidden_size=1024, num_attention_heads=16)
assert cfg.hidden_size % cfg.num_attention_heads == 0

assert cfg.hidden_size % cfg.num_attention_heads == 0, 'invalid head config'

When it happens

Trigger: Building the NLVR model from a BertConfig where hidden_size % num_attention_heads != 0 (e.g. hidden_size=1024 with num_attention_heads=48), typically from an edited config.json.

Common situations: Custom NLVR experiments resizing hidden_size without updating heads; copy-paste of base config into a large model; typos in num_attention_heads.

Related errors

• The hidden size (%d) is not a multiple of the number of atte
• checkpoint url or path is invalid
• checkpoint url or path is invalid
• Wrong shape for input_ids (shape {}) or attention_mask (shap
• You cannot specify both input_ids and inputs_embeds at the s

AI-assisted analysis of lllyasviel/Fooocus@ae05379cc9 (2026-08-15). Data as JSON: /api/errors/c2216bc5ae77f0cb. Report an issue: GitHub.