Lightning-AI/pytorch-lightning · error · TypeError

The given dataset must implement the `__len__` method.

Error message

The given dataset must implement the `__len__` method.

What it means

Raised by the DistributedSampler wrapper's __init__ in Lightning's overrides/distributed.py when the wrapped dataset is not Sized (does not implement __len__). Distributed sampling needs the dataset length to partition indices evenly across ranks, so unlengthed datasets cannot be distributed this way.

Source

Thrown at src/lightning/pytorch/overrides/distributed.py:195

class UnrepeatedDistributedSampler(DistributedSampler):
    """A fork of the PyTorch DistributedSampler that doesn't repeat data, instead allowing the number of batches per
    process to be off-by-one from each other. This makes this sampler usable for predictions (it's deterministic and
    doesn't require shuffling). It is potentially unsafe to use this sampler for training, because during training the
    DistributedDataParallel syncs buffers on each forward pass, so it could freeze if one of the processes runs one
    fewer batch. During prediction, buffers are only synced on the first batch, so this is safe to use as long as each
    process runs at least one batch. We verify this in an assert.

    Taken from https://github.com/jpuigcerver/PyLaia/blob/v1.0.0/laia/data/unpadded_distributed_sampler.py and
    https://github.com/pytorch/pytorch/issues/25162#issuecomment-634146002

    """

    def __init__(self, *args: Any, **kwargs: Any) -> None:
        super().__init__(*args, **kwargs)
        if not isinstance(self.dataset, Sized):
            raise TypeError("The given dataset must implement the `__len__` method.")
        self.num_samples = len(range(self.rank, len(self.dataset), self.num_replicas))
        self.total_size = len(self.dataset)
        # If any process has at least one batch, every other process needs to
        # have at least one batch, or the DistributedDataParallel could lock up.
        assert self.num_samples >= 1 or self.total_size == 0

    @override
    def __iter__(self) -> Iterator[list[int]]:
        if not isinstance(self.dataset, Sized):
            raise TypeError("The given dataset must implement the `__len__` method.")
        if self.shuffle:
            # deterministically shuffle based on epoch
            g = torch.Generator()
            g.manual_seed(self.epoch)
            indices = torch.randperm(len(self.dataset), generator=g).tolist()
        else:
            indices = list(range(len(self.dataset)))

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Implement `__len__` on the dataset returning the true number of samples
2. For true streams, use a length-aware iterable strategy (e.g. wrap with a length-estimating dataset or use Lightning's IterableDataset with LightningDataModule configured for distributed iteration)
3. Switch to a map-style dataset backed by an index file

Example fix

# before
class StreamDataset(IterableDataset):
    def __iter__(self):
        yield from self.source  # no __len__

# after
class StreamDataset(IterableDataset):
    def __len__(self):
        return self.source.approx_length  # required for distributed sampling
    def __iter__(self):
        yield from self.source

from collections.abc import Sized

assert isinstance(dataset, Sized) and callable(getattr(dataset, '__len__', None)), \
    'dataset must implement __len__ for distributed training'

def dataset_is_sized(ds) -> bool:
    return isinstance(ds, Sized)

When it happens

Trigger: Using a distributed strategy with a plain IterableDataset (no __len__) where Lightning/PyTorch wraps the dataset for distributed training; datasets built from generators or streams passed to DDP training.

Common situations: Streaming datasets (Kafka, TFRecords, large web scrapes) used with DDP without Lightning's special iterable handling; custom Dataset subclasses where __len__ was forgotten.

Related errors

• `{type(self).__name__}` does not own a group. HINT: try `col
• `{type(self).__name__}` already owns a group.
• `{type(self).__name__}` does not own a group to destroy.
• Torch distributed is not available.
• Unsupported op {op!r} of type {type(op).__name__}

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/0b8664053df07325. Report an issue: GitHub.