rohitg00/ai-engineering-from-scratch · error · ValueError

max_tokens must be greater than 1

Error message

max_tokens must be greater than 1

What it means

Error "max_tokens must be greater than 1" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/43-hdf5-tokenized-corpus/code/main.py:423

                    self.dropped_lines += 1
                    continue
                yield value


def pack_documents(
    tokenizer: Tokenizer,
    documents: Iterable[str],
    max_tokens: int,
) -> Iterator[list[int]]:
    """Pack tokenized documents into fixed-length groups with boundary tokens.

    Yields lists of exactly max_tokens token ids. Long documents are split
    across groups; short documents share a group separated by BOUNDARY_TOKEN_ID.
    The final group may be shorter than max_tokens and is yielded as-is.
    """

    if max_tokens <= 1:
        raise ValueError("max_tokens must be greater than 1")
    buffer: list[int] = []
    for text in documents:
        token_ids = tokenizer.encode(text)
        if buffer:
            buffer.append(BOUNDARY_TOKEN_ID)
        buffer.extend(token_ids)
        while len(buffer) >= max_tokens:
            yield buffer[:max_tokens]
            buffer = buffer[max_tokens:]
    if buffer:
        yield buffer


def tokenize_jsonl_path(
    jsonl_path: Path,
    output_dir: Path,
    shard_id: str,
    chunk_size: int = DEFAULT_CHUNK_SIZE,

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/43-hdf5-tokenized-corpus/code/main.py:423 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/8cac5bad6a38bed0. Report an issue: GitHub.