rohitg00/ai-engineering-from-scratch · error · ValueError

token and positional embeddings must share d_model

Error message

token and positional embeddings must share d_model

What it means

Error "token and positional embeddings must share d_model" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/32-token-positional-embeddings/code/main.py:148

    """

    def __init__(
        self,
        token_embedding: TokenEmbedding,
        positional_embedding: nn.Module,
    ) -> None:
        super().__init__()
        if not isinstance(token_embedding, TokenEmbedding):
            raise TypeError("token_embedding must be a TokenEmbedding")
        if not isinstance(
            positional_embedding,
            (LearnedPositionalEmbedding, SinusoidalPositionalEmbedding),
        ):
            raise TypeError(
                "positional_embedding must be Learned or Sinusoidal Positional Embedding"
            )
        if token_embedding.d_model != getattr(positional_embedding, "d_model", None):
            raise ValueError("token and positional embeddings must share d_model")
        self.token_embedding = token_embedding
        self.positional_embedding = positional_embedding

    @property
    def d_model(self) -> int:
        return self.token_embedding.d_model

    def forward(self, ids: torch.Tensor) -> torch.Tensor:
        if ids.dim() != 2:
            raise ValueError(f"ids must be (B, T), got shape {tuple(ids.shape)}")
        seq_len = ids.shape[1]
        tok = self.token_embedding(ids)
        pos = self.positional_embedding(seq_len)
        return tok + pos.unsqueeze(0)


def count_parameters(module: nn.Module) -> int:
    return sum(p.numel() for p in module.parameters() if p.requires_grad)

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/32-token-positional-embeddings/code/main.py:148 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/4483d1abb4f117f3. Report an issue: GitHub.