{"record":{"id":"eb244c69229a1bbb","repo":"open-mmlab/mmdetection","slug":"the-hidden-size-config-hidden-size-is-not-a-mu","errorCode":null,"errorMessage":"The hidden size ({config.hidden_size}) is not a multiple of the number of attention heads ({config.num_attention_heads})","messagePattern":"The hidden size \\((.+?)\\) is not a multiple of the number of attention heads \\((.+?)\\)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"mmdet/models/utils/vlfuse_helper.py","lineNumber":565,"sourceCode":"        config (:class:`~transformers.BertConfig`):\n            The configuration object that\n            contains various parameters for the model.\n        clamp_min_for_underflow (bool, optional):\n            Whether to clamp the minimum value of the hidden states\n             to prevent underflow. Defaults to `False`.\n        clamp_max_for_overflow (bool, optional):\n            Whether to clamp the maximum value of the hidden states\n            to prevent overflow. Defaults to `False`.\n    \"\"\"\n\n    def __init__(self,\n                 config: BertConfig,\n                 clamp_min_for_underflow: bool = False,\n                 clamp_max_for_overflow: bool = False):\n        super().__init__()\n        if config.hidden_size % config.num_attention_heads != 0 and \\\n                not hasattr(config, 'embedding_size'):\n            raise ValueError(f'The hidden size ({config.hidden_size}) is '\n                             'not a multiple of the number of attention '\n                             f'heads ({config.num_attention_heads})')\n\n        self.num_attention_heads = config.num_attention_heads\n        self.attention_head_size = int(config.hidden_size /\n                                       config.num_attention_heads)\n        self.all_head_size = self.num_attention_heads * \\\n            self.attention_head_size\n\n        self.query = nn.Linear(config.hidden_size, self.all_head_size)\n        self.key = nn.Linear(config.hidden_size, self.all_head_size)\n        self.value = nn.Linear(config.hidden_size, self.all_head_size)\n\n        self.dropout = nn.Dropout(config.attention_probs_dropout_prob)\n        self.position_embedding_type = getattr(config,\n                                               'position_embedding_type',\n                                               'absolute')\n        if self.position_embedding_type == 'relative_key' or \\","sourceCodeStart":547,"sourceCodeEnd":583,"githubUrl":"https://github.com/open-mmlab/mmdetection/blob/cfd5d3a985b0249de009b67d04f37263e11cdf3d/mmdet/models/utils/vlfuse_helper.py#L547-L583","documentation":"BertSelfAttention-style attention in GLIP/grounding configs requires hidden_size divisible by num_attention_heads so each head gets an integer head size. This check is copied from HuggingFace BERT; configs with embedding_size (ALBERT-style) are exempt.","triggerScenarios":"Loading a modified BERT text encoder config where hidden_size % num_attention_heads != 0 and no embedding_size attribute is present, e.g. hidden_size=768 with num_attention_heads=14.","commonSituations":"Custom/trimmed text backbones for GLIP/GroundingRCNN, hand-edited bert-base-uncased config.json values, or a config pulled from an incompatible transformers version.","solutions":["Fix the text encoder config so hidden_size is divisible by num_attention_heads (e.g. 768/12)","Load a stock bert config instead of hand-editing","If porting ALBERT-like models, keep embedding_size set (the code exempts it)"],"exampleFix":"# before\nBertConfig(hidden_size=768, num_attention_heads=14)\n# after\nBertConfig(hidden_size=768, num_attention_heads=12)","handlingStrategy":"validation","validationCode":"assert config.hidden_size % config.num_attention_heads == 0 or hasattr(config, 'embedding_size')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use stock BERT configs for GLIP text encoders","Add a sanity check in config loading for text backbones"],"tags":["mmdetection","bert","attention","config-validation"],"backgroundTag":"transformer-dimension-mismatch","analyzedSha":"cfd5d3a985b0249de009b67d04f37263e11cdf3d","analyzedAt":"2026-08-27T20:54:20.183Z","schemaVersion":2},"datasetVersion":"2026-08-28T00:17:15.603Z"}