lllyasviel/Fooocus · error · ValueError

The hidden size (%d) is not a multiple of the number of atte

Error message

The hidden size (%d) is not a multiple of the number of attention heads (%d)

What it means

BertSelfAttention.__init__ validates that hidden_size divides evenly by num_attention_heads, because each head gets hidden_size/num_attention_heads dimensions for the Q/K/V projections. If not divisible (and no legacy embedding_size attribute exists), multi-head split is impossible and the model raises immediately at construction time.

Source

Thrown at extras/BLIP/models/med.py:102

        if inputs_embeds is None:
            inputs_embeds = self.word_embeddings(input_ids)

        embeddings = inputs_embeds

        if self.position_embedding_type == "absolute":
            position_embeddings = self.position_embeddings(position_ids)
            embeddings += position_embeddings
        embeddings = self.LayerNorm(embeddings)
        embeddings = self.dropout(embeddings)
        return embeddings


class BertSelfAttention(nn.Module):
    def __init__(self, config, is_cross_attention):
        super().__init__()
        self.config = config
        if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, "embedding_size"):
            raise ValueError(
                "The hidden size (%d) is not a multiple of the number of attention "
                "heads (%d)" % (config.hidden_size, config.num_attention_heads)
            )
        
        self.num_attention_heads = config.num_attention_heads
        self.attention_head_size = int(config.hidden_size / config.num_attention_heads)
        self.all_head_size = self.num_attention_heads * self.attention_head_size

        self.query = nn.Linear(config.hidden_size, self.all_head_size)
        if is_cross_attention:
            self.key = nn.Linear(config.encoder_width, self.all_head_size)
            self.value = nn.Linear(config.encoder_width, self.all_head_size)
        else:
            self.key = nn.Linear(config.hidden_size, self.all_head_size)
            self.value = nn.Linear(config.hidden_size, self.all_head_size)

        self.dropout = nn.Dropout(config.attention_probs_dropout_prob)
        self.position_embedding_type = getattr(config, "position_embedding_type", "absolute")

View on GitHub (pinned to ae05379cc9)

Solutions

1. Set num_attention_heads to a divisor of hidden_size (e.g. 768 -> 12 or 24 heads)
2. If you changed hidden_size, recompute heads so hidden_size % num_attention_heads == 0
3. Validate the config before model construction and fail fast with a clear message

Example fix

// before
config = BertConfig(hidden_size=768, num_attention_heads=10)

// after
config = BertConfig(hidden_size=768, num_attention_heads=12)
assert config.hidden_size % config.num_attention_heads == 0

def validate_bert_config(cfg):
    assert cfg.hidden_size % cfg.num_attention_heads == 0, (
        f'hidden_size {cfg.hidden_size} not divisible by num_attention_heads {cfg.num_attention_heads}')

When it happens

Trigger: Constructing a BertConfig with e.g. hidden_size=768, num_attention_heads=10 (768 % 10 != 0), then building any BLIP MED (BERT) model; typically from a hand-edited config.json or a from_pretrained with overridden config values.

Common situations: Custom model sizing experiments that change hidden_size or num_attention_heads independently; typos in config files (num_attention_heads=14 instead of 12); porting configs between model sizes (base vs large).

Related errors

• The hidden size (%d) is not a multiple of the number of atte
• checkpoint url or path is invalid
• Wrong shape for input_ids (shape {}) or attention_mask (shap
• You cannot specify both input_ids and inputs_embeds at the s
• You have to specify either input_ids or inputs_embeds or enc

AI-assisted analysis of lllyasviel/Fooocus@ae05379cc9 (2026-08-15). Data as JSON: /api/errors/685dd4c042c28485. Report an issue: GitHub.