rohitg00/ai-engineering-from-scratch · error · ValueError

chunk_size must be positive

Error message

chunk_size must be positive

What it means

Error "chunk_size must be positive" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/43-hdf5-tokenized-corpus/code/main.py:117

        byte_ids = [int(i) - self.BYTE_OFFSET for i in ids if int(i) >= self.BYTE_OFFSET]
        return bytes(byte_ids).decode("utf-8", errors="replace")


class HDF5ShardWriter:
    """Stream tokens into a resizable HDF5 dataset with chunk-sized buffering.

    Open in a `with` block to guarantee the residual buffer is flushed and the
    closing attributes (token_count, sha256) are written.
    """

    def __init__(
        self,
        path: Path,
        chunk_size: int = DEFAULT_CHUNK_SIZE,
        dataset_name: str = "tokens",
    ) -> None:
        if chunk_size <= 0:
            raise ValueError("chunk_size must be positive")
        self.path = Path(path)
        self.chunk_size = chunk_size
        self.dataset_name = dataset_name
        self._buffer: list[int] = []
        self._token_count = 0
        self._document_count = 0
        self._hasher = hashlib.sha256()
        self._file: h5py.File | None = None
        self._dataset: h5py.Dataset | None = None

    def __enter__(self) -> "HDF5ShardWriter":
        self._file = h5py.File(self.path, "w", libver="latest")
        self._dataset = self._file.create_dataset(
            self.dataset_name,
            shape=(0,),
            maxshape=(None,),
            chunks=(self.chunk_size,),
            dtype=TOKEN_DTYPE,

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/43-hdf5-tokenized-corpus/code/main.py:117 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/88194c68e158508a. Report an issue: GitHub.