rohitg00/ai-engineering-from-scratch · error · KeyError

unknown token id: {token_id}

Error message

unknown token id: {token_id}

What it means

Error "unknown token id: {token_id}" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/30-bpe-tokenizer-from-scratch/code/main.py:228

    return out


def _encode_pretokenized(tokenizer: BPETokenizer, text: str) -> list[int]:
    out: list[int] = []
    for chunk in _pretokenize(text):
        out.extend(_encode_chunk(tokenizer, chunk))
    return out


def decode(tokenizer: BPETokenizer, ids: list[int]) -> str:
    """Decode `ids` back to a string. Inverse of `encode` for round-trip safe input."""
    pieces: list[bytes] = []
    for token_id in ids:
        if token_id in tokenizer.id_to_special:
            pieces.append(tokenizer.id_to_special[token_id].encode("utf-8"))
            continue
        if token_id not in tokenizer.vocab:
            raise KeyError(f"unknown token id: {token_id}")
        pieces.append(tokenizer.vocab[token_id])
    return b"".join(pieces).decode("utf-8", errors="replace")


def save(tokenizer: BPETokenizer, path: str) -> None:
    """Serialize the tokenizer to a JSON file."""
    payload = {
        "vocab": {
            str(token_id): list(token_bytes)
            for token_id, token_bytes in tokenizer.vocab.items()
        },
        "merges": [
            [list(pair), new_id]
            for pair, new_id in tokenizer.merges.items()
        ],
        "specials": tokenizer.special_to_id,
    }
    with open(path, "w", encoding="utf-8") as f:

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/30-bpe-tokenizer-from-scratch/code/main.py:228 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/11fb6a782ec8ca43. Report an issue: GitHub.