rohitg00/ai-engineering-from-scratch · error · ValueError

batch_size ({cfg.batch_size}) cannot exceed corpus size ({le

Error message

batch_size ({cfg.batch_size}) cannot exceed corpus size ({len(corpus)}) with replace=False

What it means

Error "batch_size ({cfg.batch_size}) cannot exceed corpus size ({len(corpus)}) with replace=False" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/62-vision-language-pretraining/code/main.py:235

            ids[j] = 1 + (base + j * 3 + (i % 5)) % (vocab_size - 1)
        pairs.append((img, torch.from_numpy(ids).unsqueeze(0)))
    return pairs


def sample_batch(pairs: list[tuple[torch.Tensor, torch.Tensor]], indices: list[int]
                 ) -> tuple[torch.Tensor, torch.Tensor]:
    imgs = torch.cat([pairs[i][0] for i in indices], dim=0)
    ids = torch.cat([pairs[i][1] for i in indices], dim=0)
    return imgs, ids


def train(cfg: PretrainConfig) -> dict:
    torch.manual_seed(cfg.seed)
    model = MultimodalModel(cfg).train()
    opt = torch.optim.Adam(model.parameters(), lr=cfg.lr)
    corpus = make_mock_corpus(cfg.seed + 1, cfg.n_pairs, cfg.text_vocab, cfg.max_text_len)
    if cfg.batch_size > len(corpus):
        raise ValueError(
            f"batch_size ({cfg.batch_size}) cannot exceed corpus size ({len(corpus)}) "
            "with replace=False"
        )

    rng = np.random.default_rng(cfg.seed + 2)
    history = {"contrast": [], "lm": [], "total": []}

    for step in range(cfg.steps):
        idx = rng.choice(len(corpus), size=cfg.batch_size, replace=False).tolist()
        imgs, ids = sample_batch(corpus, idx)
        contrast, lm, stats = model(imgs, ids)
        total = contrast + cfg.lm_weight * lm
        opt.zero_grad(set_to_none=True)
        total.backward()
        opt.step()

        history["contrast"].append(contrast.item())
        history["lm"].append(lm.item())

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/62-vision-language-pretraining/code/main.py:235 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/97390995cbe92300. Report an issue: GitHub.