lancedb/lancedb · error · TypeError

StreamingDataLoader does not support StreamingDataset…

Error message

StreamingDataLoader does not support StreamingDataset subclasses that override __iter__ because they cannot provide exact per-yield checkpoint state

What it means

StreamingDataLoader checkpoints state after every yield; this only works if iteration goes through StreamingDataset.__iter__ exactly. Subclasses that override __iter__ could yield samples the loader cannot track, so they are rejected at construction time.

Solutions

Remove the __iter__ override from the StreamingDataset subclass and use the base class iteration (customize behavior via dataset parameters like shuffle or filter instead)
If custom iteration is essential, use plain torch DataLoader and forgo exact per-yield checkpointing
Wrap filtering logic in the dataset's __getitem__/transform rather than __iter__

Example fix

// before
class MyDS(StreamingDataset):
    def __iter__(self):
        for x in super().__iter__():
            if x.keep: yield x
// after
ds = StreamingDataset(uri, filter=keep_fn)  # no __iter__ override
loader = StreamingDataLoader(ds, batch_size=32)

Defensive patterns

Strategy: type-guard

Validate before calling

assert type(ds).__iter__ is StreamingDataset.__iter__, 'subclass overrides __iter__'

Type guard

from lancedb.streaming import StreamingDataset
def is_exact_iterator(ds):
    return isinstance(ds, StreamingDataset) and type(ds).__iter__ is StreamingDataset.__iter__

Try / catch

try:
    loader = StreamingDataLoader(ds, batch_size=32)
except TypeError:
    raise RuntimeError('remove the __iter__ override from your StreamingDataset subclass to use StreamingDataLoader')

Prevention

Never override __iter__ on StreamingDataset subclasses used with checkpointing
Express filtering/shuffling via dataset options, not iteration overrides
Test loader construction in CI for any custom dataset subclass

When it happens

Trigger: def class MyDataset(StreamingDataset): def __iter__(self): ... and passing MyDataset to StreamingDataLoader; any subclass that customizes iteration order or filters yields.

Common situations: Users subclassing StreamingDataset to add filtering, shuffling, or custom yield logic; codebases with an existing __iter__ override that worked with plain DataLoader.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of lancedb/lancedb@c7b051aff7 (2026-09-08). Data as JSON: /api/errors/2336fe9a5a6ef1fe. Report an issue: GitHub.

Appendix: source

Thrown at python/python/lancedb/streaming.py:1791

    ``dataset`` must be a
    [StreamingDataset][lancedb.streaming.StreamingDataset].
    Subclasses that override ``StreamingDataset.__iter__`` are not supported
    because the custom iterator cannot provide the exact per-yield checkpoint
    snapshots required by this loader.

    Examples
    --------
    >>> # dataset = StreamingDataset(table, num_splits=2)
    >>> # loader = StreamingDataLoader(dataset, batch_size=8, num_workers=2)
    >>> # batch = next(iter(loader))
    >>> # checkpoint = dataset.state_dict()
    """

    def __init__(self, dataset: StreamingDataset, *args, **kwargs):
        if not isinstance(dataset, StreamingDataset):
            raise TypeError("StreamingDataLoader requires a StreamingDataset")
        if type(dataset).__iter__ is not StreamingDataset.__iter__:
            raise TypeError(
                "StreamingDataLoader does not support StreamingDataset subclasses "
                "that override __iter__ because they cannot provide exact "
                "per-yield checkpoint state"
            )
        if kwargs.get("in_order", True) is False:
            raise ValueError(
                "StreamingDataLoader requires in_order=True for deterministic "
                "consumer checkpoints"
            )
        if kwargs.get("persistent_workers", False):
            raise ValueError(
                "StreamingDataLoader does not support persistent_workers=True "
                "because worker prefetch state cannot be reset from a checkpoint"
            )
        self._streaming_dataset = dataset
        super().__init__(_StreamingDatasetAdapter(dataset), *args, **kwargs)
        if self.drop_last:
            raise ValueError(

View on GitHub (pinned to c7b051aff7)