rohitg00/ai-engineering-from-scratch · error · ValueError

logits must be 3D and targets 2D, got {logits.shape} {target

Error message

logits must be 3D and targets 2D, got {logits.shape} {target_ids.shape}

What it means

Error "logits must be 3D and targets 2D, got {logits.shape} {target_ids.shape}" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/62-vision-language-pretraining/code/main.py:107

    scale = log_tau.exp().clamp(min=1e-3, max=100.0)
    sim = (img_n @ txt_n.T) * scale

    targets = torch.arange(n, device=sim.device)
    loss_i2t = F.cross_entropy(sim, targets)
    loss_t2i = F.cross_entropy(sim.T, targets)
    return (loss_i2t + loss_t2i) * 0.5, sim


def lm_loss(logits: torch.Tensor, target_ids: torch.Tensor,
            padding_id: int = PAD_ID) -> torch.Tensor:
    """Next-token cross-entropy with padding masked.

    `logits` shape is (B, L, V). `target_ids` shape is (B, L). The shift is
    applied outside this function so the caller controls which positions are
    predictions and which are inputs.
    """
    if logits.dim() != 3 or target_ids.dim() != 2:
        raise ValueError(f"logits must be 3D and targets 2D, got {logits.shape} {target_ids.shape}")
    b, l, v = logits.shape
    flat_logits = logits.reshape(b * l, v)
    flat_target = target_ids.reshape(b * l)
    return F.cross_entropy(flat_logits, flat_target, ignore_index=padding_id)


class TextSideEncoder(nn.Module):
    """Tiny text encoder: embedding lookup + mean pool over non-padding tokens."""

    def __init__(self, vocab_size: int, embed_dim: int) -> None:
        super().__init__()
        self.embed = nn.Embedding(vocab_size, embed_dim, padding_idx=PAD_ID)

    def forward(self, ids: torch.Tensor) -> torch.Tensor:
        if ids.dim() != 2:
            raise ValueError(f"expected (B, L), got {tuple(ids.shape)}")
        x = self.embed(ids)
        mask = (ids != PAD_ID).float().unsqueeze(-1)

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/62-vision-language-pretraining/code/main.py:107 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/1851beb9ea70e133. Report an issue: GitHub.