WZMIAOMIAO/deep-learning-for-image-processing · error · ValueError

Embedding dim must be divisible by number of heads in {}. Go

Error message

Embedding dim must be divisible by number of heads in {}. Got: embed_dim={} and num_heads={}

What it means

MultiHeadSelfAttention requires embed_dim to be exactly divisible by num_heads so per-head dimensions are equal (qkv_proj projects to 3*embed_dim and is split per head). A non-divisible pair makes the head split produce uneven chunks, so it is rejected in __init__.

Source

Thrown at pytorch_classification/MobileViT/transformer.py:37

    Shape:
        - Input: :math:`(N, P, C_{in})` where :math:`N` is batch size, :math:`P` is number of patches,
        and :math:`C_{in}` is input embedding dim
        - Output: same shape as the input

    """

    def __init__(
        self,
        embed_dim: int,
        num_heads: int,
        attn_dropout: float = 0.0,
        bias: bool = True,
        *args,
        **kwargs
    ) -> None:
        super().__init__()
        if embed_dim % num_heads != 0:
            raise ValueError(
                "Embedding dim must be divisible by number of heads in {}. Got: embed_dim={} and num_heads={}".format(
                    self.__class__.__name__, embed_dim, num_heads
                )
            )

        self.qkv_proj = nn.Linear(in_features=embed_dim, out_features=3 * embed_dim, bias=bias)

        self.attn_dropout = nn.Dropout(p=attn_dropout)
        self.out_proj = nn.Linear(in_features=embed_dim, out_features=embed_dim, bias=bias)

        self.head_dim = embed_dim // num_heads
        self.scaling = self.head_dim ** -0.5
        self.softmax = nn.Softmax(dim=-1)
        self.num_heads = num_heads
        self.embed_dim = embed_dim

    def forward(self, x_q: Tensor) -> Tensor:
        # [N, P, C]

View on GitHub (pinned to 1ec3fe6f37)

Solutions

  1. Set num_heads to a divisor of embed_dim (e.g. 4 or 8 for common dims).
  2. Adjust embed_dim/transformer_dim to a multiple of num_heads.
  3. Compute head_dim first and assert embed_dim % head_dim == 0 in your config builder.

Example fix

// before
attn = MultiHeadSelfAttention(embed_dim=192, num_heads=8)  # 192 % 8 == 24, fine; bad case: 200 % 8 != 0
// after
attn = MultiHeadSelfAttention(embed_dim=200, num_heads=5)  # or embed_dim=192, num_heads=8
Defensive patterns

Strategy: validation

Validate before calling

def check_attention_cfg(embed_dim: int, num_heads: int):
    if embed_dim % num_heads != 0:
        raise ValueError(f"embed_dim {embed_dim} must be divisible by num_heads {num_heads}")

Type guard

def heads_divide(embed_dim: int, num_heads: int) -> bool:
    return isinstance(embed_dim, int) and isinstance(num_heads, int) and num_heads > 0 and embed_dim % num_heads == 0

Try / catch

try:
    attn = MultiHeadSelfAttention(dim=embed_dim, num_heads=num_heads)
except ValueError as e:
    print(e)
    num_heads = max(h for h in range(1, num_heads + 1) if embed_dim % h == 0)
    attn = MultiHeadSelfAttention(dim=embed_dim, num_heads=num_heads)

Prevention

When it happens

Trigger: Constructing the attention block with e.g. embed_dim=100, num_heads=6, or a MobileViT cfg where transformer_dim is not a multiple of num_heads.

Common situations: Hand-tuned model width for edge deployment, copied configs with mismatched head counts, changing num_heads without recomputing transformer_dim.

Related errors

AI-assisted analysis of WZMIAOMIAO/deep-learning-for-image-processing@1ec3fe6f37 (2026-08-30). Data as JSON: /api/errors/a6e5f6a9e26bea0b. Report an issue: GitHub.