rohitg00/ai-engineering-from-scratch · error · ValueError

hidden {self.hidden} not divisible by heads {self.heads}

Error message

hidden {self.hidden} not divisible by heads {self.heads}

What it means

Error "hidden {self.hidden} not divisible by heads {self.heads}" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/61-cross-attention-fusion/code/main.py:41

import torch.nn.functional as F


@dataclass(frozen=True)
class DecoderConfig:
    hidden: int = 256
    heads: int = 8
    depth: int = 4
    mlp_ratio: float = 4.0
    text_vocab: int = 1024
    max_text_len: int = 32
    vision_dim: int = 256
    vision_tokens: int = 197
    dropout: float = 0.0

    @property
    def head_dim(self) -> int:
        if self.hidden % self.heads != 0:
            raise ValueError(f"hidden {self.hidden} not divisible by heads {self.heads}")
        return self.hidden // self.heads


def causal_mask(length: int) -> torch.Tensor:
    """Lower-triangular boolean mask of shape (length, length).

    Cell [i, j] is True if token i may attend to token j (j <= i).
    """
    return torch.tril(torch.ones(length, length, dtype=torch.bool))


class CausalSelfAttention(nn.Module):
    def __init__(self, cfg: DecoderConfig) -> None:
        super().__init__()
        self.cfg = cfg
        self.qkv = nn.Linear(cfg.hidden, cfg.hidden * 3, bias=True)
        self.out = nn.Linear(cfg.hidden, cfg.hidden, bias=True)
        self.drop = nn.Dropout(cfg.dropout)

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/61-cross-attention-fusion/code/main.py:41 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/0fcc6a35803e3984. Report an issue: GitHub.