rohitg00/ai-engineering-from-scratch · error · ValueError

vocab_size must be > 50, got {vocab_size}

Error message

vocab_size must be > 50, got {vocab_size}

What it means

Error "vocab_size must be > 50, got {vocab_size}" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/62-vision-language-pretraining/code/main.py:201

            logits = self.caption_logits(memory, inputs)
            lm = lm_loss(logits, targets, padding_id=PAD_ID)

        diag = sim.diag().mean().item()
        offdiag = (sim.sum() - sim.diag().sum()).item() / max(1, b * b - b)
        stats = {"diag": diag, "off_diag": offdiag, "tau": self.log_tau.exp().item()}
        return contrast, lm, stats


def make_mock_corpus(seed: int, n_pairs: int, vocab_size: int, max_len: int
                     ) -> list[tuple[torch.Tensor, torch.Tensor]]:
    """Build a deterministic mock corpus of n_pairs synthetic image-caption pairs.

    Caption tokens are correlated with the image seed so the model has a small
    amount of learnable signal across the contrastive batch. Token id 0 is
    reserved for padding.
    """
    if vocab_size <= 50:
        raise ValueError(f"vocab_size must be > 50, got {vocab_size}")
    pairs = []
    rng = np.random.default_rng(seed)
    for i in range(n_pairs):
        img_seed = seed * 100 + i
        rng_i = np.random.default_rng(img_seed)
        noise = rng_i.standard_normal((3, 32, 32)).astype("float32") * 0.2
        gx, gy = np.meshgrid(np.linspace(0.0, 1.0, 32), np.linspace(0.0, 1.0, 32))
        bias = (i % 7) / 7.0
        img = np.clip(noise + bias, -1.0, 1.0).astype("float32")
        img = torch.from_numpy(img).unsqueeze(0)

        length = min(6 + (i % 8), max_len)
        ids = np.zeros((max_len,), dtype=np.int64)
        base = (i * 17) % (vocab_size - 50)
        for j in range(length):
            ids[j] = 1 + (base + j * 3 + (i % 5)) % (vocab_size - 1)
        pairs.append((img, torch.from_numpy(ids).unsqueeze(0)))
    return pairs

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/62-vision-language-pretraining/code/main.py:201 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/728d9d2a480ad3e1. Report an issue: GitHub.