{"record":{"id":"03c59d445c61a755","repo":"huggingface/transformers","slug":"the-embed-dim-self-embed-dim-is-not-a-multiple","errorCode":null,"errorMessage":"The embed_dim ({self.embed_dim}) is not a multiple of the number of attention heads ({self.num_heads}).","messagePattern":"The embed_dim \\((.+?)\\) is not a multiple of the number of attention heads \\((.+?)\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/configuration_utils.py","lineNumber":503,"sourceCode":"        if self.output_attentions and self._attn_implementation not in [\"eager\", None]:\n            raise ValueError(\n                \"The `output_attentions` attribute is not supported when using the `attn_implementation` set to \"\n                f\"{self._attn_implementation}. Please set it to 'eager' instead.\"\n            )\n\n    def validate_architecture(self):\n        \"\"\"Part of `@strict`-powered validation. Validates the architecture of the config.\"\"\"\n        if self.is_heterogeneous:\n            for config in self.per_layer_config:\n                config.validate_architecture()\n            return\n        if (\n            hasattr(self, \"head_dim\")\n            and hasattr(self, \"num_heads\")\n            and hasattr(self, \"embed_dim\")\n            and self.head_dim * self.num_heads != self.embed_dim\n        ):\n            raise ValueError(\n                f\"The embed_dim ({self.embed_dim}) is not a multiple of the number of attention \"\n                f\"heads ({self.num_heads}).\"\n            )\n\n    def validate_token_ids(self):\n        \"\"\"Part of `@strict`-powered validation. Validates the contents of the special tokens.\"\"\"\n        text_config = self.get_text_config(decoder=True)\n        vocab_size = getattr(text_config, \"vocab_size\", None)\n        if vocab_size is not None:\n            # Check for all special tokens, e..g. pad_token_id, image_token_id, audio_token_id\n            for name in text_config:\n                value = getattr(text_config, name)\n                if name.endswith(\"_token_id\") and isinstance(value, int) and not 0 <= value < vocab_size:\n                    # Can't be an exception until we can load configs that fail validation: several configs on the Hub\n                    # store invalid special tokens, e.g. `pad_token_id=-1`\n                    logger.warning_once(\n                        f\"Model config: {name} must be `None` or an integer within the vocabulary (between 0 \"\n                        f\"and {vocab_size - 1}), got {value}. This may result in unexpected behavior.\"","sourceCodeStart":485,"sourceCodeEnd":521,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/configuration_utils.py#L485-L521","documentation":"ValueError from validate_architecture (part of @strict config validation) when the config defines head_dim, num_heads and embed_dim and head_dim * num_heads != embed_dim. The model would build a projection matrix of the wrong shape, so the mismatch is caught at config time; heterogeneous configs recurse into each per-layer config.","triggerScenarios":"Constructing a config with embed_dim=512, num_heads=8, head_dim=80 (80*8=640 != 512); editing one of the three fields (e.g. num_heads for a variant) without rebalancing the others; wrong values parsed from a foreign checkpoint.","commonSituations":"Architecture search scripts mutating num_heads or head_dim independently; converting weights from another framework that names dims differently; typos in config YAMLs.","solutions":["Make head_dim * num_heads equal embed_dim, e.g. set head_dim = embed_dim // num_heads","If you intended a different embed_dim, change embed_dim to head_dim * num_heads consistently","Validate the triple right after parsing any user-provided architecture spec"],"exampleFix":"# before\ncfg = MyConfig(embed_dim=512, num_heads=8, head_dim=80)\n# after\ncfg = MyConfig(embed_dim=512, num_heads=8, head_dim=64)","handlingStrategy":"validation","validationCode":"assert 'head_dim' not in params or params['head_dim'] * params['num_heads'] == params['embed_dim'], \\\n    'head_dim * num_heads must equal embed_dim'","typeGuard":"def dims_consistent(embed_dim: int, num_heads: int, head_dim: int) -> bool:\n    return head_dim * num_heads == embed_dim","tryCatchPattern":"try:\n    cfg.validate_architecture()\nexcept ValueError as e:\n    if 'not a multiple' in str(e):\n        params['head_dim'] = params['embed_dim'] // params['num_heads']\n        cfg = MyConfig(**params)\n    else:\n        raise","preventionTips":["Compute head_dim as embed_dim // num_heads instead of hardcoding it","Validate the dims triple whenever a config comes from search tooling or external YAML"],"tags":["config","architecture","validation"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}