rohitg00/ai-engineering-from-scratch · error · ValueError

causal mask shape {tuple(mask.shape)} does not match (n, n)

Error message

causal mask shape {tuple(mask.shape)} does not match (n, n) = ({n}, {n})

What it means

Error "causal mask shape {tuple(mask.shape)} does not match (n, n) = ({n}, {n})" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/61-cross-attention-fusion/code/main.py:71

class CausalSelfAttention(nn.Module):
    def __init__(self, cfg: DecoderConfig) -> None:
        super().__init__()
        self.cfg = cfg
        self.qkv = nn.Linear(cfg.hidden, cfg.hidden * 3, bias=True)
        self.out = nn.Linear(cfg.hidden, cfg.hidden, bias=True)
        self.drop = nn.Dropout(cfg.dropout)
        self.scale = 1.0 / math.sqrt(cfg.head_dim)

    def forward(self, x: torch.Tensor, mask: torch.Tensor | None = None) -> torch.Tensor:
        b, n, d = x.shape
        h, hd = self.cfg.heads, self.cfg.head_dim
        qkv = self.qkv(x).reshape(b, n, 3, h, hd).permute(2, 0, 3, 1, 4)
        q, k, v = qkv[0], qkv[1], qkv[2]

        scores = (q @ k.transpose(-2, -1)) * self.scale
        if mask is not None:
            if mask.shape != (n, n):
                raise ValueError(
                    f"causal mask shape {tuple(mask.shape)} does not match (n, n) = ({n}, {n})"
                )
            scores = scores.masked_fill(~mask.unsqueeze(0).unsqueeze(0), float("-inf"))
        attn = F.softmax(scores, dim=-1)
        out = (attn @ v).transpose(1, 2).reshape(b, n, d)
        return self.drop(self.out(out))


class CrossAttention(nn.Module):
    """Multi-head cross-attention.

    Query comes from text tokens; key and value come from image memory.
    Supports a kv_cache argument so the projection of image memory can be
    computed once and reused across decode steps.
    """

    def __init__(self, cfg: DecoderConfig) -> None:
        super().__init__()

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/61-cross-attention-fusion/code/main.py:71 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/5ff7f535cdf85df2. Report an issue: GitHub.