{"record":{"id":"0b8664053df07325","repo":"Lightning-AI/pytorch-lightning","slug":"the-given-dataset-must-implement-the-len-met","errorCode":null,"errorMessage":"The given dataset must implement the `__len__` method.","messagePattern":"The given dataset must implement the `__len__` method\\.","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"src/lightning/pytorch/overrides/distributed.py","lineNumber":195,"sourceCode":"\n\nclass UnrepeatedDistributedSampler(DistributedSampler):\n    \"\"\"A fork of the PyTorch DistributedSampler that doesn't repeat data, instead allowing the number of batches per\n    process to be off-by-one from each other. This makes this sampler usable for predictions (it's deterministic and\n    doesn't require shuffling). It is potentially unsafe to use this sampler for training, because during training the\n    DistributedDataParallel syncs buffers on each forward pass, so it could freeze if one of the processes runs one\n    fewer batch. During prediction, buffers are only synced on the first batch, so this is safe to use as long as each\n    process runs at least one batch. We verify this in an assert.\n\n    Taken from https://github.com/jpuigcerver/PyLaia/blob/v1.0.0/laia/data/unpadded_distributed_sampler.py and\n    https://github.com/pytorch/pytorch/issues/25162#issuecomment-634146002\n\n    \"\"\"\n\n    def __init__(self, *args: Any, **kwargs: Any) -> None:\n        super().__init__(*args, **kwargs)\n        if not isinstance(self.dataset, Sized):\n            raise TypeError(\"The given dataset must implement the `__len__` method.\")\n        self.num_samples = len(range(self.rank, len(self.dataset), self.num_replicas))\n        self.total_size = len(self.dataset)\n        # If any process has at least one batch, every other process needs to\n        # have at least one batch, or the DistributedDataParallel could lock up.\n        assert self.num_samples >= 1 or self.total_size == 0\n\n    @override\n    def __iter__(self) -> Iterator[list[int]]:\n        if not isinstance(self.dataset, Sized):\n            raise TypeError(\"The given dataset must implement the `__len__` method.\")\n        if self.shuffle:\n            # deterministically shuffle based on epoch\n            g = torch.Generator()\n            g.manual_seed(self.epoch)\n            indices = torch.randperm(len(self.dataset), generator=g).tolist()\n        else:\n            indices = list(range(len(self.dataset)))\n","sourceCodeStart":177,"sourceCodeEnd":213,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/pytorch/overrides/distributed.py#L177-L213","documentation":"Raised by the DistributedSampler wrapper's __init__ in Lightning's overrides/distributed.py when the wrapped dataset is not Sized (does not implement __len__). Distributed sampling needs the dataset length to partition indices evenly across ranks, so unlengthed datasets cannot be distributed this way.","triggerScenarios":"Using a distributed strategy with a plain IterableDataset (no __len__) where Lightning/PyTorch wraps the dataset for distributed training; datasets built from generators or streams passed to DDP training.","commonSituations":"Streaming datasets (Kafka, TFRecords, large web scrapes) used with DDP without Lightning's special iterable handling; custom Dataset subclasses where __len__ was forgotten.","solutions":["Implement `__len__` on the dataset returning the true number of samples","For true streams, use a length-aware iterable strategy (e.g. wrap with a length-estimating dataset or use Lightning's IterableDataset with LightningDataModule configured for distributed iteration)","Switch to a map-style dataset backed by an index file"],"exampleFix":"# before\nclass StreamDataset(IterableDataset):\n    def __iter__(self):\n        yield from self.source  # no __len__\n\n# after\nclass StreamDataset(IterableDataset):\n    def __len__(self):\n        return self.source.approx_length  # required for distributed sampling\n    def __iter__(self):\n        yield from self.source","handlingStrategy":"type-guard","validationCode":"from collections.abc import Sized\n\nassert isinstance(dataset, Sized) and callable(getattr(dataset, '__len__', None)), \\\n    'dataset must implement __len__ for distributed training'","typeGuard":"def dataset_is_sized(ds) -> bool:\n    return isinstance(ds, Sized)","tryCatchPattern":null,"preventionTips":["Implement __len__ on custom datasets as a matter of contract","For streams, pre-compute or estimate length and expose it via __len__ before distributed runs"],"tags":["pytorch-lightning","distributed","distributed-sampler","iterable-dataset"],"backgroundTag":"iterable-dataset-length-unsupported","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}