rohitg00/ai-engineering-from-scratch · error · ValueError

sequence length {seq} exceeds context length {self.cfg.conte

Error message

sequence length {seq} exceeds context length {self.cfg.context_length}

What it means

Error "sequence length {seq} exceeds context length {self.cfg.context_length}" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/35-gpt-model-assembly/code/main.py:162

    def _init_weights(self, module: nn.Module) -> None:
        if isinstance(module, nn.Linear):
            nn.init.normal_(module.weight, mean=0.0, std=0.02)
            if module.bias is not None:
                nn.init.zeros_(module.bias)
        elif isinstance(module, nn.Embedding):
            nn.init.normal_(module.weight, mean=0.0, std=0.02)

    def _scale_residual_projections(self) -> None:
        scale = 1.0 / math.sqrt(2 * self.cfg.num_layers)
        for block in self.blocks:
            block.attn.out_proj.weight.data.mul_(scale)
            block.mlp.fc2.weight.data.mul_(scale)

    def forward(self, tokens: torch.Tensor) -> torch.Tensor:
        batch, seq = tokens.shape
        if seq > self.cfg.context_length:
            raise ValueError(
                f"sequence length {seq} exceeds context length {self.cfg.context_length}"
            )
        tok = self.tok_embed(tokens)
        pos = self.pos_embed(self.position_ids[:seq])
        x = self.embed_dropout(tok + pos)
        for block in self.blocks:
            x = block(x)
        x = self.final_ln(x)
        logits = self.lm_head(x)
        return logits


def count_parameters(model: nn.Module) -> int:
    """Count unique parameters. Weight tied tensors are counted once."""
    seen: dict[int, int] = {}
    for param in model.parameters():
        seen[id(param)] = param.numel()
    return sum(seen.values())

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/35-gpt-model-assembly/code/main.py:162 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/11b2be8fffb9dd8a. Report an issue: GitHub.