{"record":{"id":"a6e5f6a9e26bea0b","repo":"WZMIAOMIAO/deep-learning-for-image-processing","slug":"embedding-dim-must-be-divisible-by-number-of-heads","errorCode":null,"errorMessage":"Embedding dim must be divisible by number of heads in {}. Got: embed_dim={} and num_heads={}","messagePattern":"Embedding dim must be divisible by number of heads in (.+?)\\. Got: embed_dim=(.+?) and num_heads=(.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"pytorch_classification/MobileViT/transformer.py","lineNumber":37,"sourceCode":"    Shape:\n        - Input: :math:`(N, P, C_{in})` where :math:`N` is batch size, :math:`P` is number of patches,\n        and :math:`C_{in}` is input embedding dim\n        - Output: same shape as the input\n\n    \"\"\"\n\n    def __init__(\n        self,\n        embed_dim: int,\n        num_heads: int,\n        attn_dropout: float = 0.0,\n        bias: bool = True,\n        *args,\n        **kwargs\n    ) -> None:\n        super().__init__()\n        if embed_dim % num_heads != 0:\n            raise ValueError(\n                \"Embedding dim must be divisible by number of heads in {}. Got: embed_dim={} and num_heads={}\".format(\n                    self.__class__.__name__, embed_dim, num_heads\n                )\n            )\n\n        self.qkv_proj = nn.Linear(in_features=embed_dim, out_features=3 * embed_dim, bias=bias)\n\n        self.attn_dropout = nn.Dropout(p=attn_dropout)\n        self.out_proj = nn.Linear(in_features=embed_dim, out_features=embed_dim, bias=bias)\n\n        self.head_dim = embed_dim // num_heads\n        self.scaling = self.head_dim ** -0.5\n        self.softmax = nn.Softmax(dim=-1)\n        self.num_heads = num_heads\n        self.embed_dim = embed_dim\n\n    def forward(self, x_q: Tensor) -> Tensor:\n        # [N, P, C]","sourceCodeStart":19,"sourceCodeEnd":55,"githubUrl":"https://github.com/WZMIAOMIAO/deep-learning-for-image-processing/blob/1ec3fe6f374fc9969973a61f819de25658595afa/pytorch_classification/MobileViT/transformer.py#L19-L55","documentation":"MultiHeadSelfAttention requires embed_dim to be exactly divisible by num_heads so per-head dimensions are equal (qkv_proj projects to 3*embed_dim and is split per head). A non-divisible pair makes the head split produce uneven chunks, so it is rejected in __init__.","triggerScenarios":"Constructing the attention block with e.g. embed_dim=100, num_heads=6, or a MobileViT cfg where transformer_dim is not a multiple of num_heads.","commonSituations":"Hand-tuned model width for edge deployment, copied configs with mismatched head counts, changing num_heads without recomputing transformer_dim.","solutions":["Set num_heads to a divisor of embed_dim (e.g. 4 or 8 for common dims).","Adjust embed_dim/transformer_dim to a multiple of num_heads.","Compute head_dim first and assert embed_dim % head_dim == 0 in your config builder."],"exampleFix":"// before\nattn = MultiHeadSelfAttention(embed_dim=192, num_heads=8)  # 192 % 8 == 24, fine; bad case: 200 % 8 != 0\n// after\nattn = MultiHeadSelfAttention(embed_dim=200, num_heads=5)  # or embed_dim=192, num_heads=8","handlingStrategy":"validation","validationCode":"def check_attention_cfg(embed_dim: int, num_heads: int):\n    if embed_dim % num_heads != 0:\n        raise ValueError(f\"embed_dim {embed_dim} must be divisible by num_heads {num_heads}\")","typeGuard":"def heads_divide(embed_dim: int, num_heads: int) -> bool:\n    return isinstance(embed_dim, int) and isinstance(num_heads, int) and num_heads > 0 and embed_dim % num_heads == 0","tryCatchPattern":"try:\n    attn = MultiHeadSelfAttention(dim=embed_dim, num_heads=num_heads)\nexcept ValueError as e:\n    print(e)\n    num_heads = max(h for h in range(1, num_heads + 1) if embed_dim % h == 0)\n    attn = MultiHeadSelfAttention(dim=embed_dim, num_heads=num_heads)","preventionTips":["Use standard head counts (4, 8, 16) with dims that are their multiples","Check divisibility in your config builder before model instantiation","Keep embed_dim and num_heads defined together in one config object"],"tags":["pytorch","transformer","attention","divisibility"],"backgroundTag":"embed-dim-head-mismatch","analyzedSha":"1ec3fe6f374fc9969973a61f819de25658595afa","analyzedAt":"2026-08-30T09:19:11.901Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}