open-mmlab/mmdetection · info

Byte tensor for attn_mask is deprecated.

Error message

Byte tensor for attn_mask is deprecated.                     Use bool tensor instead.

What it means

MultiheadAttention warns that a uint8 (byte) attention mask is deprecated and converts it to bool. PyTorch deprecated byte masks because nonzero bytes are interpreted inconsistently; bool masks are the supported format.

Source

Thrown at mmdet/models/layers/transformer/utils.py:564

        assert head_dims * self.num_heads == hidden_dims, \
            f'{"hidden_dims must be divisible by num_heads"}'
        scaling = float(head_dims)**-0.5

        q = query * scaling
        k = key
        v = value

        if attn_mask is not None:
            assert attn_mask.dtype == torch.float32 or \
                   attn_mask.dtype == torch.float64 or \
                   attn_mask.dtype == torch.float16 or \
                   attn_mask.dtype == torch.uint8 or \
                   attn_mask.dtype == torch.bool, \
                   'Only float, byte, and bool types are supported for \
                    attn_mask'

            if attn_mask.dtype == torch.uint8:
                warnings.warn('Byte tensor for attn_mask is deprecated.\
                     Use bool tensor instead.')
                attn_mask = attn_mask.to(torch.bool)
            if attn_mask.dim() == 2:
                attn_mask = attn_mask.unsqueeze(0)
                if list(attn_mask.size()) != [1, query.size(1), key.size(1)]:
                    raise RuntimeError(
                        'The size of the 2D attn_mask is not correct.')
            elif attn_mask.dim() == 3:
                if list(attn_mask.size()) != [
                        bs * self.num_heads,
                        query.size(1),
                        key.size(1)
                ]:
                    raise RuntimeError(
                        'The size of the 3D attn_mask is not correct.')
            else:
                raise RuntimeError(
                    "attn_mask's dimension {} is not supported".format(

View on GitHub (pinned to cfd5d3a985)

Solutions

1. Create masks with dtype torch.bool (mask.to(torch.bool) or direct bool construction)
2. Ensure padding/key_padding masks come from utils as bool tensors

Example fix

# before
attn_mask = (positions == pad).byte()
# after
attn_mask = (positions == pad).bool()

assert attn_mask.dtype == torch.bool, f'expected bool mask, got {attn_mask.dtype}'

def is_bool_mask(m):
    return isinstance(m, torch.Tensor) and m.dtype == torch.bool

When it happens

Trigger: Passing attn_mask with dtype torch.uint8 to mmdet's MultiheadAttention forward path (e.g. from custom DETR-style code or legacy checkpoints/pipelines).

Common situations: Code written for old PyTorch where byte masks were common; masks created via .byte() instead of .bool().

Related errors

• sampler should be an instance of ``Sampler``, but got {sampl
• pretrained must be a str or None
• The size of the 2D attn_mask is not correct.
• The size of the 3D attn_mask is not correct.
• attn_mask's dimension {} is not supported

AI-assisted analysis of open-mmlab/mmdetection@cfd5d3a985 (2026-08-27). Data as JSON: /api/errors/53c348cfd5248db0. Report an issue: GitHub.