WZMIAOMIAO/deep-learning-for-image-processing · error · ValueError

Transformer input dimension should be divisible by head dime

Error message

Transformer input dimension should be divisible by head dimension. Got {} and {}.

What it means

MobileViT builds its transformer attention with head_dim = transformer_dim // num_heads, then asserts transformer_dim is divisible by head_dim. This can only fail when num_heads does not divide transformer_dim evenly (integer truncation makes head_dim wrong), so the check catches misconfigured channel/head combinations early.

Source

Thrown at pytorch_classification/MobileViT/model.py:489

        if stride == 2:
            layer = InvertedResidual(
                in_channels=input_channel,
                out_channels=cfg.get("out_channels"),
                stride=stride,
                expand_ratio=cfg.get("mv_expand_ratio", 4)
            )

            block.append(layer)
            input_channel = cfg.get("out_channels")

        transformer_dim = cfg["transformer_channels"]
        ffn_dim = cfg.get("ffn_dim")
        num_heads = cfg.get("num_heads", 4)
        head_dim = transformer_dim // num_heads

        if transformer_dim % head_dim != 0:
            raise ValueError("Transformer input dimension should be divisible by head dimension. "
                             "Got {} and {}.".format(transformer_dim, head_dim))

        block.append(MobileViTBlock(
            in_channels=input_channel,
            transformer_dim=transformer_dim,
            ffn_dim=ffn_dim,
            n_transformer_blocks=cfg.get("transformer_blocks", 1),
            patch_h=cfg.get("patch_h", 2),
            patch_w=cfg.get("patch_w", 2),
            dropout=cfg.get("dropout", 0.1),
            ffn_dropout=cfg.get("ffn_dropout", 0.0),
            attn_dropout=cfg.get("attn_dropout", 0.1),
            head_dim=head_dim,
            conv_ksize=3
        ))

        return nn.Sequential(*block), input_channel

View on GitHub (pinned to 1ec3fe6f37)

Solutions

1. Choose transformer_channels divisible by num_heads (default 4).
2. Set num_heads in the cfg to a divisor of transformer_channels.
3. Verify head_dim = transformer_dim // num_heads leaves no remainder before building the model.

Example fix

// before
cfg = {"transformer_channels": 200, "num_heads": 8}
// after
cfg = {"transformer_channels": 200, "num_heads": 4}  # 200 % 50 == 0

def check_vit_cfg(cfg):
    td = cfg["transformer_channels"]
    nh = cfg.get("num_heads", 4)
    assert td % nh == 0, f"transformer_channels {td} not divisible by num_heads {nh}"

def heads_divide(transformer_dim: int, num_heads: int) -> bool:
    return num_heads > 0 and transformer_dim % num_heads == 0

try:
    model = mobile_vit_xxs(cfg)
except ValueError as e:
    print(e)
    cfg["num_heads"] = next(h for h in (4, 2, 1) if cfg["transformer_channels"] % h == 0)
    model = mobile_vit_xxs(cfg)

When it happens

Trigger: Configuring a MobileViT variant cfg where transformer_channels is not an integer multiple of num_heads, e.g. transformer_channels=200 with the default num_heads=4 leaving head_dim=50... or values like transformer_dim=100, num_heads=8 (head_dim=12, 100 % 12 != 0).

Common situations: Hand-editing the model config dict to shrink the model for mobile deployment while keeping default num_heads; porting configs between MobileViT variants.

Related errors

• Embedding dim must be divisible by number of heads in {}. Go
• not support data format '{self.data_format}'
• illegal stride value.
• image: {} isn't RGB mode.
• image: {} isn't RGB mode.

AI-assisted analysis of WZMIAOMIAO/deep-learning-for-image-processing@1ec3fe6f37 (2026-08-30). Data as JSON: /api/errors/2745444326e20ceb. Report an issue: GitHub.