rohitg00/ai-engineering-from-scratch · error · ValueError

seq_len {t} exceeds max_context_length {self.max_context_len

Error message

seq_len {t} exceeds max_context_length {self.max_context_length}

What it means

Error "seq_len {t} exceeds max_context_length {self.max_context_length}" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/33-multihead-self-attention/code/main.py:74

        b, t, _ = x.shape
        return x.view(b, t, self.n_heads, self.d_head).transpose(1, 2)

    def _merge_heads(self, x: torch.Tensor) -> torch.Tensor:
        b, h, t, dh = x.shape
        return x.transpose(1, 2).contiguous().view(b, t, h * dh)

    def forward(
        self,
        x: torch.Tensor,
        return_weights: bool = False,
    ) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]:
        if x.dim() != 3:
            raise ValueError(f"input must be (B, T, D), got shape {tuple(x.shape)}")
        b, t, d = x.shape
        if d != self.d_model:
            raise ValueError(f"feature dim {d} != d_model {self.d_model}")
        if t > self.max_context_length:
            raise ValueError(f"seq_len {t} exceeds max_context_length {self.max_context_length}")

        qkv = self.qkv_proj(x)
        q, k, v = qkv.chunk(3, dim=-1)

        q = self._split_heads(q)
        k = self._split_heads(k)
        v = self._split_heads(v)

        scores = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.d_head)
        mask_slice = self.causal_mask[:t, :t]
        scores = scores.masked_fill(mask_slice == 0, float("-inf"))

        weights = F.softmax(scores, dim=-1)
        weights = self.attn_dropout(weights)

        context = torch.matmul(weights, v)
        context = self._merge_heads(context)

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/33-multihead-self-attention/code/main.py:74 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/2198a674469ededc. Report an issue: GitHub.