rohitg00/ai-engineering-from-scratch · error · ValueError

at least one shard entry is required

Error message

at least one shard entry is required

What it means

Error "at least one shard entry is required" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/43-hdf5-tokenized-corpus/code/main.py:265

            "total_tokens": running_offset,
            "shards": [entry.to_dict() for entry in entries],
        }
        index_path.write_text(json.dumps(body, sort_keys=True, indent=2), encoding="utf-8")
        return entries


class MmapTokenStore:
    """Memory-mapped read access to a sharded HDF5 token corpus.

    The store opens each shard file once in SWMR mode. A request for
    `get_slice(start, stop)` is routed across shards and the result is returned
    as a flat NumPy uint16 array. Reads land in the page cache; the dataloader
    pays one copy when it crosses into a training tensor.
    """

    def __init__(self, shard_entries: list[ShardIndexEntry]) -> None:
        if not shard_entries:
            raise ValueError("at least one shard entry is required")
        self._entries = shard_entries
        self._files: list[h5py.File] = []
        self._datasets: list[h5py.Dataset] = []
        try:
            for entry in shard_entries:
                self._files.append(h5py.File(entry.path, "r", swmr=True))
            self._datasets = [f["tokens"] for f in self._files]
        except Exception:
            for opened in self._files:
                try:
                    opened.close()
                except Exception:
                    pass
            self._files = []
            self._datasets = []
            raise
        self._total_tokens = sum(entry.token_count for entry in shard_entries)

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/43-hdf5-tokenized-corpus/code/main.py:265 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/cf14b45a790387d3. Report an issue: GitHub.