rohitg00/ai-engineering-from-scratch · error · RuntimeError

slice read produced {cursor} tokens, expected {stop - start}

Error message

slice read produced {cursor} tokens, expected {stop - start}

What it means

Error "slice read produced {cursor} tokens, expected {stop - start}" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/43-hdf5-tokenized-corpus/code/main.py:327

            return np.empty((0,), dtype=TOKEN_DTYPE)
        out = np.empty((stop - start,), dtype=TOKEN_DTYPE)
        cursor = 0
        for entry, dataset in zip(self._entries, self._datasets):
            shard_start = entry.global_start
            shard_stop = shard_start + entry.token_count
            if stop <= shard_start:
                break
            if start >= shard_stop:
                continue
            local_start = max(0, start - shard_start)
            local_stop = min(entry.token_count, stop - shard_start)
            length = local_stop - local_start
            if length <= 0:
                continue
            out[cursor : cursor + length] = dataset[local_start:local_stop]
            cursor += length
        if cursor != stop - start:
            raise RuntimeError(
                f"slice read produced {cursor} tokens, expected {stop - start}"
            )
        return out


class SlidingWindowDataloader:
    """Random sliding-window sampler over a flat token stream."""

    def __init__(
        self,
        store: MmapTokenStore,
        window_size: int = DEFAULT_WINDOW_SIZE,
        batch_size: int = 4,
        seed: int = 0,
    ) -> None:
        if window_size <= 1:
            raise ValueError("window_size must be greater than 1")
        if batch_size <= 0:

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/43-hdf5-tokenized-corpus/code/main.py:327 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/21dfcdfdcf2992e2. Report an issue: GitHub.