{"record":{"id":"c2216bc5ae77f0cb","repo":"lllyasviel/Fooocus","slug":"the-hidden-size-d-is-not-a-multiple-of-the-numb-c2216b","errorCode":null,"errorMessage":"The hidden size (%d) is not a multiple of the number of attention heads (%d)","messagePattern":"The hidden size \\((.+?)\\) is not a multiple of the number of attention heads \\((.+?)\\)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"extras/BLIP/models/nlvr_encoder.py","lineNumber":92,"sourceCode":"        if inputs_embeds is None:\n            inputs_embeds = self.word_embeddings(input_ids)\n\n        embeddings = inputs_embeds\n\n        if self.position_embedding_type == \"absolute\":\n            position_embeddings = self.position_embeddings(position_ids)\n            embeddings += position_embeddings\n        embeddings = self.LayerNorm(embeddings)\n        embeddings = self.dropout(embeddings)\n        return embeddings\n\n\nclass BertSelfAttention(nn.Module):\n    def __init__(self, config, is_cross_attention):\n        super().__init__()\n        self.config = config\n        if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, \"embedding_size\"):\n            raise ValueError(\n                \"The hidden size (%d) is not a multiple of the number of attention \"\n                \"heads (%d)\" % (config.hidden_size, config.num_attention_heads)\n            )\n        \n        self.num_attention_heads = config.num_attention_heads\n        self.attention_head_size = int(config.hidden_size / config.num_attention_heads)\n        self.all_head_size = self.num_attention_heads * self.attention_head_size\n\n        self.query = nn.Linear(config.hidden_size, self.all_head_size)\n        if is_cross_attention:\n            self.key = nn.Linear(config.encoder_width, self.all_head_size)\n            self.value = nn.Linear(config.encoder_width, self.all_head_size)\n        else:\n            self.key = nn.Linear(config.hidden_size, self.all_head_size)\n            self.value = nn.Linear(config.hidden_size, self.all_head_size)\n\n        self.dropout = nn.Dropout(config.attention_probs_dropout_prob)\n        self.position_embedding_type = getattr(config, \"position_embedding_type\", \"absolute\")","sourceCodeStart":74,"sourceCodeEnd":110,"githubUrl":"https://github.com/lllyasviel/Fooocus/blob/ae05379cc97bc4361ec8b4ec90193dab21be763f/extras/BLIP/models/nlvr_encoder.py#L74-L110","documentation":"Same constructor validation as the MED BertSelfAttention but in the NLVR twin-encoder copy (nlvr_encoder.py): hidden_size must be divisible by num_attention_heads so each head receives an integer slice of the embedding for Q/K/V projections; otherwise model construction fails immediately.","triggerScenarios":"Building the NLVR model from a BertConfig where hidden_size % num_attention_heads != 0 (e.g. hidden_size=1024 with num_attention_heads=48), typically from an edited config.json.","commonSituations":"Custom NLVR experiments resizing hidden_size without updating heads; copy-paste of base config into a large model; typos in num_attention_heads.","solutions":["Fix the config: choose num_attention_heads that divides hidden_size","Recompute both values together when scaling the model","Validate config invariants before instantiating the model"],"exampleFix":"// before\ncfg = BertConfig(hidden_size=1024, num_attention_heads=48)\n\n// after\ncfg = BertConfig(hidden_size=1024, num_attention_heads=16)\nassert cfg.hidden_size % cfg.num_attention_heads == 0","handlingStrategy":"validation","validationCode":"assert cfg.hidden_size % cfg.num_attention_heads == 0, 'invalid head config'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Validate NLVR config before model construction","Keep hidden_size/num_attention_heads changes paired","Add a config-lint step to CI"],"tags":["blip","nlvr","bert","config","attention"],"backgroundTag":null,"analyzedSha":"ae05379cc97bc4361ec8b4ec90193dab21be763f","analyzedAt":"2026-08-15T04:23:59.533Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}