rohitg00/ai-engineering-from-scratch · error · ValueError

seq_len {seq_len} exceeds max_context_length {self.max_conte

Error message

seq_len {seq_len} exceeds max_context_length {self.max_context_length}

What it means

Error "seq_len {seq_len} exceeds max_context_length {self.max_context_length}" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/33-multihead-self-attention/code/main.py:139

        if d_model < 1:
            raise ValueError(f"d_model must be >= 1, got {d_model}")
        if d_model % 2 != 0:
            raise ValueError(f"d_model must be even, got {d_model}")
        self.max_context_length = max_context_length
        pos = torch.arange(max_context_length, dtype=torch.float32).unsqueeze(1)
        i = torch.arange(d_model // 2, dtype=torch.float32)
        denom = base ** (2 * i / d_model)
        angle = pos / denom
        pe = torch.zeros(max_context_length, d_model, dtype=torch.float32)
        pe[:, 0::2] = torch.sin(angle)
        pe[:, 1::2] = torch.cos(angle)
        self.register_buffer("pe", pe, persistent=False)

    def forward(self, seq_len: int) -> torch.Tensor:
        if seq_len < 1:
            raise ValueError(f"seq_len must be >= 1, got {seq_len}")
        if seq_len > self.max_context_length:
            raise ValueError(
                f"seq_len {seq_len} exceeds max_context_length {self.max_context_length}"
            )
        return self.pe[:seq_len]


class TinyAttentionLM(nn.Module):
    """Embedding + attention + LM head. Just enough to train a copy task."""

    def __init__(
        self,
        vocab_size: int,
        d_model: int,
        n_heads: int,
        max_context_length: int,
    ) -> None:
        super().__init__()
        self.token_emb = TokenEmbedding(vocab_size, d_model)
        self.pos_emb = SinusoidalPositionalEmbedding(max_context_length, d_model)

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/33-multihead-self-attention/code/main.py:139 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/9d7cf882da9ce829. Report an issue: GitHub.