open-mmlab/mmdetection · error · RuntimeError

attn_mask's dimension {} is not supported

Error message

attn_mask's dimension {} is not supported

What it means

MultiheadAttention.forward_attn only supports attn_mask of dimension 2 or 3; a 1D or 4D+ mask raises RuntimeError with the offending dimension in the message.

Source

Thrown at mmdet/models/layers/transformer/utils.py:581

            if attn_mask.dtype == torch.uint8:
                warnings.warn('Byte tensor for attn_mask is deprecated.\
                     Use bool tensor instead.')
                attn_mask = attn_mask.to(torch.bool)
            if attn_mask.dim() == 2:
                attn_mask = attn_mask.unsqueeze(0)
                if list(attn_mask.size()) != [1, query.size(1), key.size(1)]:
                    raise RuntimeError(
                        'The size of the 2D attn_mask is not correct.')
            elif attn_mask.dim() == 3:
                if list(attn_mask.size()) != [
                        bs * self.num_heads,
                        query.size(1),
                        key.size(1)
                ]:
                    raise RuntimeError(
                        'The size of the 3D attn_mask is not correct.')
            else:
                raise RuntimeError(
                    "attn_mask's dimension {} is not supported".format(
                        attn_mask.dim()))
        # attn_mask's dim is 3 now.

        if key_padding_mask is not None and key_padding_mask.dtype == int:
            key_padding_mask = key_padding_mask.to(torch.bool)

        q = q.contiguous().view(bs, tgt_len, self.num_heads,
                                head_dims).permute(0, 2, 1, 3).flatten(0, 1)
        if k is not None:
            k = k.contiguous().view(bs, src_len, self.num_heads,
                                    head_dims).permute(0, 2, 1,
                                                       3).flatten(0, 1)
        if v is not None:
            v = v.contiguous().view(bs, src_len, self.num_heads,
                                    v_head_dims).permute(0, 2, 1,
                                                         3).flatten(0, 1)

View on GitHub (pinned to cfd5d3a985)

Solutions

1. If 4D [bs, heads, q, k], flatten batch and heads: attn_mask.view(bs*heads, q, k)
2. If 1D padding mask, pass it as key_padding_mask instead of attn_mask
3. Reshape/choose a 2D [q, k] or 3D [bs*heads, q, k] representation

Example fix

# before
attn_mask = torch.zeros(bs, heads, q, k)
# after
attn_mask = torch.zeros(bs, heads, q, k).view(bs*heads, q, k)

assert attn_mask.dim() in (2, 3), attn_mask.dim()

def valid_attn_mask(m): return m.dim() in (2, 3)

When it happens

Trigger: Passing a flattened boolean vector as attn_mask, or a 4D [bs, heads, q, k] mask from another codebase directly.

Common situations: Interfacing with HuggingFace-style 4D attention masks; passing key_padding-like 1D masks in the attn_mask slot.

Related errors

• The size of the 2D attn_mask is not correct.
• The size of the 3D attn_mask is not correct.
• Unknown pos_tensor shape(-1):{}
• Byte tensor for attn_mask is deprecated.
• The annotation file of Open Images Challenge should be a txt

AI-assisted analysis of open-mmlab/mmdetection@cfd5d3a985 (2026-08-27). Data as JSON: /api/errors/16bd2c40f3b440b7. Report an issue: GitHub.