{"record":{"id":"685dd4c042c28485","repo":"lllyasviel/Fooocus","slug":"the-hidden-size-d-is-not-a-multiple-of-the-numb","errorCode":null,"errorMessage":"The hidden size (%d) is not a multiple of the number of attention heads (%d)","messagePattern":"The hidden size \\((.+?)\\) is not a multiple of the number of attention heads \\((.+?)\\)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"extras/BLIP/models/med.py","lineNumber":102,"sourceCode":"        if inputs_embeds is None:\n            inputs_embeds = self.word_embeddings(input_ids)\n\n        embeddings = inputs_embeds\n\n        if self.position_embedding_type == \"absolute\":\n            position_embeddings = self.position_embeddings(position_ids)\n            embeddings += position_embeddings\n        embeddings = self.LayerNorm(embeddings)\n        embeddings = self.dropout(embeddings)\n        return embeddings\n\n\nclass BertSelfAttention(nn.Module):\n    def __init__(self, config, is_cross_attention):\n        super().__init__()\n        self.config = config\n        if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, \"embedding_size\"):\n            raise ValueError(\n                \"The hidden size (%d) is not a multiple of the number of attention \"\n                \"heads (%d)\" % (config.hidden_size, config.num_attention_heads)\n            )\n        \n        self.num_attention_heads = config.num_attention_heads\n        self.attention_head_size = int(config.hidden_size / config.num_attention_heads)\n        self.all_head_size = self.num_attention_heads * self.attention_head_size\n\n        self.query = nn.Linear(config.hidden_size, self.all_head_size)\n        if is_cross_attention:\n            self.key = nn.Linear(config.encoder_width, self.all_head_size)\n            self.value = nn.Linear(config.encoder_width, self.all_head_size)\n        else:\n            self.key = nn.Linear(config.hidden_size, self.all_head_size)\n            self.value = nn.Linear(config.hidden_size, self.all_head_size)\n\n        self.dropout = nn.Dropout(config.attention_probs_dropout_prob)\n        self.position_embedding_type = getattr(config, \"position_embedding_type\", \"absolute\")","sourceCodeStart":84,"sourceCodeEnd":120,"githubUrl":"https://github.com/lllyasviel/Fooocus/blob/ae05379cc97bc4361ec8b4ec90193dab21be763f/extras/BLIP/models/med.py#L84-L120","documentation":"BertSelfAttention.__init__ validates that hidden_size divides evenly by num_attention_heads, because each head gets hidden_size/num_attention_heads dimensions for the Q/K/V projections. If not divisible (and no legacy embedding_size attribute exists), multi-head split is impossible and the model raises immediately at construction time.","triggerScenarios":"Constructing a BertConfig with e.g. hidden_size=768, num_attention_heads=10 (768 % 10 != 0), then building any BLIP MED (BERT) model; typically from a hand-edited config.json or a from_pretrained with overridden config values.","commonSituations":"Custom model sizing experiments that change hidden_size or num_attention_heads independently; typos in config files (num_attention_heads=14 instead of 12); porting configs between model sizes (base vs large).","solutions":["Set num_attention_heads to a divisor of hidden_size (e.g. 768 -> 12 or 24 heads)","If you changed hidden_size, recompute heads so hidden_size % num_attention_heads == 0","Validate the config before model construction and fail fast with a clear message"],"exampleFix":"// before\nconfig = BertConfig(hidden_size=768, num_attention_heads=10)\n\n// after\nconfig = BertConfig(hidden_size=768, num_attention_heads=12)\nassert config.hidden_size % config.num_attention_heads == 0","handlingStrategy":"validation","validationCode":"def validate_bert_config(cfg):\n    assert cfg.hidden_size % cfg.num_attention_heads == 0, (\n        f'hidden_size {cfg.hidden_size} not divisible by num_attention_heads {cfg.num_attention_heads}')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Validate config invariants immediately after loading/creating BertConfig","When scaling hidden_size, update num_attention_heads in the same change","Add a config unit test for head-divisibility"],"tags":["blip","bert","config","attention","model-construction"],"backgroundTag":null,"analyzedSha":"ae05379cc97bc4361ec8b4ec90193dab21be763f","analyzedAt":"2026-08-15T04:23:59.533Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}