rohitg00/ai-engineering-from-scratch · error · ValueError

d_model must be >= 1, got {d_model}

Error message

d_model must be >= 1, got {d_model}

What it means

Error "d_model must be >= 1, got {d_model}" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/32-token-positional-embeddings/code/main.py:40

DEFAULT_INIT_STD = 0.02


def _init_normal(weight: torch.Tensor, std: float = DEFAULT_INIT_STD) -> None:
    """Init a parameter tensor in place from a small Gaussian."""
    with torch.no_grad():
        weight.normal_(mean=0.0, std=std)


class TokenEmbedding(nn.Module):
    """Vocabulary-id to vector lookup."""

    def __init__(self, vocab_size: int, d_model: int, init_std: float = DEFAULT_INIT_STD) -> None:
        super().__init__()
        if vocab_size < 1:
            raise ValueError(f"vocab_size must be >= 1, got {vocab_size}")
        if d_model < 1:
            raise ValueError(f"d_model must be >= 1, got {d_model}")
        self.vocab_size = vocab_size
        self.d_model = d_model
        self.embedding = nn.Embedding(vocab_size, d_model)
        _init_normal(self.embedding.weight, std=init_std)

    def forward(self, ids: torch.Tensor) -> torch.Tensor:
        if ids.dtype != torch.long:
            raise TypeError(f"ids must be long tensor, got {ids.dtype}")
        if ids.dim() != 2:
            raise ValueError(f"ids must be (B, T), got shape {tuple(ids.shape)}")
        return self.embedding(ids)


class LearnedPositionalEmbedding(nn.Module):
    """Position-id to vector lookup with learned parameters."""

    def __init__(
        self,

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/32-token-positional-embeddings/code/main.py:40 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/8c69272e30c69865. Report an issue: GitHub.