rohitg00/ai-engineering-from-scratch · error · ValueError

d_model ({cfg.d_model}) must be divisible by num_heads ({cfg

Error message

d_model ({cfg.d_model}) must be divisible by num_heads ({cfg.num_heads})

What it means

Error "d_model ({cfg.d_model}) must be divisible by num_heads ({cfg.num_heads})" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/34-transformer-block/code/main.py:68

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        mean = x.mean(dim=-1, keepdim=True)
        var = x.var(dim=-1, keepdim=True, unbiased=False)
        return self.scale * (x - mean) / torch.sqrt(var + self.eps) + self.shift


class MultiHeadAttention(nn.Module):
    """Multi head causal self attention with a fused QKV projection.

    Fused QKV: one linear of width 3 * d_model instead of three linears, one
    kernel launch, one matmul. The causal mask is registered as a buffer so it
    is allocated once at construction and sliced per forward.
    """

    def __init__(self, cfg: BlockConfig) -> None:
        super().__init__()
        if cfg.d_model % cfg.num_heads != 0:
            raise ValueError(
                f"d_model ({cfg.d_model}) must be divisible by num_heads ({cfg.num_heads})"
            )
        self.d_model = cfg.d_model
        self.num_heads = cfg.num_heads
        self.head_dim = cfg.d_model // cfg.num_heads
        self.context_length = cfg.context_length

        self.qkv = nn.Linear(cfg.d_model, 3 * cfg.d_model, bias=cfg.use_bias)
        self.out_proj = nn.Linear(cfg.d_model, cfg.d_model, bias=cfg.use_bias)
        self.attn_dropout = nn.Dropout(cfg.attn_dropout)
        self.resid_dropout = nn.Dropout(cfg.residual_dropout)

        mask = torch.triu(
            torch.ones(cfg.context_length, cfg.context_length, dtype=torch.bool),
            diagonal=1,
        )
        self.register_buffer("causal_mask", mask, persistent=False)

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/34-transformer-block/code/main.py:68 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/b549ffafb60daa40. Report an issue: GitHub.