rohitg00/ai-engineering-from-scratch · error · ValueError

n_heads must be >= 1, got {n_heads}

Error message

n_heads must be >= 1, got {n_heads}

What it means

Error "n_heads must be >= 1, got {n_heads}" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/33-multihead-self-attention/code/main.py:35

import torch.nn.functional as F


class MultiHeadSelfAttention(nn.Module):
    """Multi-head self-attention with a single QKV linear and a causal mask."""

    def __init__(
        self,
        d_model: int,
        n_heads: int,
        max_context_length: int,
        attn_dropout: float = 0.0,
        out_dropout: float = 0.0,
    ) -> None:
        super().__init__()
        if d_model < 1:
            raise ValueError(f"d_model must be >= 1, got {d_model}")
        if n_heads < 1:
            raise ValueError(f"n_heads must be >= 1, got {n_heads}")
        if d_model % n_heads != 0:
            raise ValueError(
                f"d_model ({d_model}) must be divisible by n_heads ({n_heads})"
            )
        if max_context_length < 1:
            raise ValueError(f"max_context_length must be >= 1, got {max_context_length}")
        self.d_model = d_model
        self.n_heads = n_heads
        self.d_head = d_model // n_heads
        self.max_context_length = max_context_length

        self.qkv_proj = nn.Linear(d_model, 3 * d_model, bias=True)
        self.out_proj = nn.Linear(d_model, d_model, bias=True)
        self.attn_dropout = nn.Dropout(attn_dropout)
        self.out_dropout = nn.Dropout(out_dropout)

        causal_mask = torch.tril(torch.ones(max_context_length, max_context_length))
        self.register_buffer("causal_mask", causal_mask, persistent=False)

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/33-multihead-self-attention/code/main.py:35 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/7678b049ee05dcac. Report an issue: GitHub.