lancedb/lancedb · error · RuntimeError

StreamingDataset cannot checkpoint a standard DataLoader…

Error message

StreamingDataset cannot checkpoint a standard DataLoader with num_workers > 0 because prefetched worker progress is not consumer-committed. Use StreamingDataLoader instead.

What it means

When a plain torch DataLoader with num_workers > 0 iterates a StreamingDataset, workers prefetch batches whose progress the parent dataset cannot attribute as consumer-committed. state_dict() detects this untracked worker iteration and raises RuntimeError, because a checkpoint taken now would silently include or omit prefetched samples incorrectly. StreamingDataLoader coordinates worker progress and is the supported path.

Solutions

Switch to StreamingDataLoader(dataset, num_workers=N), which checkpoints worker-committed progress correctly.
Set num_workers=0 in the standard DataLoader so all consumption happens in the parent process.
Ensure state_dict() is called from the main process via the StreamingDataLoader's checkpoint path, not on a dataset that workers iterated directly.

Example fix

# before
loader = torch.utils.data.DataLoader(dataset, num_workers=4)
state = dataset.state_dict()
# after
from lancedb.streaming import StreamingDataLoader
loader = StreamingDataLoader(dataset, num_workers=4)
state = loader.state_dict()

Defensive patterns

Strategy: validation

Validate before calling

import torch
from torch.utils.data import DataLoader
if isinstance(loader, DataLoader) and loader.num_workers > 0 and not isinstance(loader, StreamingDataLoader):
    raise RuntimeError('use StreamingDataLoader for checkpointing with workers')

Type guard

def checkpointable(loader) -> bool:
    from lancedb.streaming import StreamingDataLoader
    return loader.num_workers == 0 or isinstance(loader, StreamingDataLoader)

Try / catch

try:
    state = dataset.state_dict()
except RuntimeError as e:
    if 'standard DataLoader with num_workers > 0' in str(e):
        raise  # cannot be recovered in-place; switch loader type first

Prevention

Use StreamingDataLoader whenever num_workers > 0 and checkpointing is needed
Call state_dict/load_state_dict through the loader, not the raw dataset
Audit training loops after migrating from stock DataLoader

When it happens

Trigger: Calling dataset.state_dict() after iterating via torch.utils.data.DataLoader(dataset, num_workers>0); the dataset flagged _untracked_worker_iteration while running inside a worker with no active parent reservation (get_worker_info() is None at checkpoint time).

Common situations: Migrating existing PyTorch training loops to StreamingDataset but keeping the stock DataLoader; toggling num_workers between runs and then checkpointing; debugging scripts that mix loader types.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of lancedb/lancedb@c7b051aff7 (2026-09-08). Data as JSON: /api/errors/be08801f35ef56f5. Report an issue: GitHub.

Appendix: source

Thrown at python/python/lancedb/streaming.py:1360

        trainer.  A standard multi-process ``DataLoader`` cannot expose that
        boundary, so calling this method after one has started raises
        ``RuntimeError`` instead of returning stale producer state.

        In row mode, the returned dict is topology-independent at global step
        boundaries. ``positions_consumed_per_split`` records how far each
        split's permutation has advanced, which can differ from the sample
        count when ``on_transform_error`` skips rows. ``StreamingDataLoader``
        combines worker state in its parent process. Combine state dicts from
        every rank with
        [merge_state_dicts][lancedb.streaming.StreamingDataset.merge_state_dicts]
        before resuming on a different topology.

        Packed state includes partial token buffers and emitted block counts
        for every logical split. When packing is sharded, merge every rank
        state with ``merge_state_dicts`` before loading it.
        """
        if self._untracked_worker_iteration[0] and get_worker_info() is None:
            raise RuntimeError(
                "StreamingDataset cannot checkpoint a standard DataLoader with "
                "num_workers > 0 because prefetched worker progress is not "
                "consumer-committed. Use StreamingDataLoader instead."
            )
        if self._checkpoint_invalid_reason is not None:
            raise RuntimeError(
                "StreamingDataset checkpointing is invalid because "
                f"{self._checkpoint_invalid_reason}. Load the last valid "
                "checkpoint into a fresh dataset before continuing."
            )
        state = self._checkpoint_snapshot()
        if self._pack_sequences is not None:
            rank_blocks = [
                state["blocks_emitted_per_split"][split] for split in self._rank_splits
            ]
            if len(set(rank_blocks)) > 1:
                raise RuntimeError(
                    "Packed StreamingDataset checkpointing is only safe at a "

View on GitHub (pinned to c7b051aff7)