{"record":{"id":"2336fe9a5a6ef1fe","repo":"lancedb/lancedb","slug":"streamingdataloader-does-not-support-streamingdata","errorCode":null,"errorMessage":"StreamingDataLoader does not support StreamingDataset subclasses that override __iter__ because they cannot provide exact per-yield checkpoint state","messagePattern":"StreamingDataLoader does not support StreamingDataset subclasses that override __iter__ because they cannot provide exact per-yield checkpoint state","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"python/python/lancedb/streaming.py","lineNumber":1791,"sourceCode":"    ``dataset`` must be a\n    [StreamingDataset][lancedb.streaming.StreamingDataset].\n    Subclasses that override ``StreamingDataset.__iter__`` are not supported\n    because the custom iterator cannot provide the exact per-yield checkpoint\n    snapshots required by this loader.\n\n    Examples\n    --------\n    >>> # dataset = StreamingDataset(table, num_splits=2)\n    >>> # loader = StreamingDataLoader(dataset, batch_size=8, num_workers=2)\n    >>> # batch = next(iter(loader))\n    >>> # checkpoint = dataset.state_dict()\n    \"\"\"\n\n    def __init__(self, dataset: StreamingDataset, *args, **kwargs):\n        if not isinstance(dataset, StreamingDataset):\n            raise TypeError(\"StreamingDataLoader requires a StreamingDataset\")\n        if type(dataset).__iter__ is not StreamingDataset.__iter__:\n            raise TypeError(\n                \"StreamingDataLoader does not support StreamingDataset subclasses \"\n                \"that override __iter__ because they cannot provide exact \"\n                \"per-yield checkpoint state\"\n            )\n        if kwargs.get(\"in_order\", True) is False:\n            raise ValueError(\n                \"StreamingDataLoader requires in_order=True for deterministic \"\n                \"consumer checkpoints\"\n            )\n        if kwargs.get(\"persistent_workers\", False):\n            raise ValueError(\n                \"StreamingDataLoader does not support persistent_workers=True \"\n                \"because worker prefetch state cannot be reset from a checkpoint\"\n            )\n        self._streaming_dataset = dataset\n        super().__init__(_StreamingDatasetAdapter(dataset), *args, **kwargs)\n        if self.drop_last:\n            raise ValueError(","sourceCodeStart":1773,"sourceCodeEnd":1809,"githubUrl":"https://github.com/lancedb/lancedb/blob/c7b051aff7039333a3f61b79217246c27676806a/python/python/lancedb/streaming.py#L1773-L1809","documentation":"StreamingDataLoader checkpoints state after every yield; this only works if iteration goes through StreamingDataset.__iter__ exactly. Subclasses that override __iter__ could yield samples the loader cannot track, so they are rejected at construction time.","triggerScenarios":"def class MyDataset(StreamingDataset): def __iter__(self): ... and passing MyDataset to StreamingDataLoader; any subclass that customizes iteration order or filters yields.","commonSituations":"Users subclassing StreamingDataset to add filtering, shuffling, or custom yield logic; codebases with an existing __iter__ override that worked with plain DataLoader.","solutions":["Remove the __iter__ override from the StreamingDataset subclass and use the base class iteration (customize behavior via dataset parameters like shuffle or filter instead)","If custom iteration is essential, use plain torch DataLoader and forgo exact per-yield checkpointing","Wrap filtering logic in the dataset's __getitem__/transform rather than __iter__"],"exampleFix":"// before\nclass MyDS(StreamingDataset):\n    def __iter__(self):\n        for x in super().__iter__():\n            if x.keep: yield x\n// after\nds = StreamingDataset(uri, filter=keep_fn)  # no __iter__ override\nloader = StreamingDataLoader(ds, batch_size=32)","handlingStrategy":"type-guard","validationCode":"assert type(ds).__iter__ is StreamingDataset.__iter__, 'subclass overrides __iter__'","typeGuard":"from lancedb.streaming import StreamingDataset\ndef is_exact_iterator(ds):\n    return isinstance(ds, StreamingDataset) and type(ds).__iter__ is StreamingDataset.__iter__","tryCatchPattern":"try:\n    loader = StreamingDataLoader(ds, batch_size=32)\nexcept TypeError:\n    raise RuntimeError('remove the __iter__ override from your StreamingDataset subclass to use StreamingDataLoader')","preventionTips":["Never override __iter__ on StreamingDataset subclasses used with checkpointing","Express filtering/shuffling via dataset options, not iteration overrides","Test loader construction in CI for any custom dataset subclass"],"tags":["python","inheritance","streaming-dataloader","unsupported-operation"],"backgroundTag":"unsupported-operation","analyzedSha":"c7b051aff7039333a3f61b79217246c27676806a","analyzedAt":"2026-09-08T23:42:37.579Z","contentChangedAt":"2026-09-08T23:42:37.579Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}