{"record":{"id":"2b1050988e821742","repo":"Lightning-AI/pytorch-lightning","slug":"all-datasets-are-iterable-style-datasets","errorCode":null,"errorMessage":"All datasets are iterable-style datasets.","messagePattern":"All datasets are iterable-style datasets\\.","errorType":"validation","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/lightning/pytorch/utilities/combined_loader.py","lineNumber":374,"sourceCode":"        \"\"\"Compute the number of batches.\"\"\"\n        if self._iterator is None:\n            raise RuntimeError(\"Please call `iter(combined_loader)` first.\")\n        return len(self._iterator)\n\n    def reset(self) -> None:\n        \"\"\"Reset the state and shutdown any workers.\"\"\"\n        if self._iterator is not None:\n            self._iterator.reset()\n            self._iterator = None\n        for iterable in self.flattened:\n            _shutdown_workers_and_reset_iterator(iterable)\n\n    def _dataset_length(self) -> int:\n        \"\"\"Compute the total length of the datasets according to the current mode.\"\"\"\n        datasets = [getattr(dl, \"dataset\", None) for dl in self.flattened]\n        lengths = [length for ds in datasets if (length := sized_len(ds)) is not None]\n        if not lengths:\n            raise NotImplementedError(\"All datasets are iterable-style datasets.\")\n        fn = _SUPPORTED_MODES[self._mode][\"fn\"]\n        return fn(lengths)\n\n    def _state_dicts(self) -> list[dict[str, Any]]:\n        \"\"\"Returns the list of state dicts for iterables in `self.flattened` that are stateful.\"\"\"\n        return [loader.state_dict() for loader in self.flattened if isinstance(loader, _Stateful)]\n\n    def _load_state_dicts(self, states: list[dict[str, Any]]) -> None:\n        \"\"\"Loads the state dicts for iterables in `self.flattened` that are stateful.\"\"\"\n        if not states:\n            return\n        stateful_loaders = [loader for loader in self.flattened if isinstance(loader, _Stateful)]\n        if len(stateful_loaders) != len(states):\n            raise RuntimeError(\n                f\"The CombinedLoader has {len(stateful_loaders)} stateful loaders, but found {len(states)} states\"\n                \" in the checkpoint. Please make sure you define the same dataloaders that were used when saving\"\n                \" the checkpoint.\"\n            )","sourceCodeStart":356,"sourceCodeEnd":392,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/pytorch/utilities/combined_loader.py#L356-L392","documentation":"CombinedLoader._dataset_length computes total dataset length by summing/reducing len() of the underlying datasets; if every dataset is iterable-style (no __len__), it cannot compute a length and raises NotImplementedError.","triggerScenarios":"Building CombinedLoader from IterableDataset-based dataloaders and triggering length computation (e.g. progress bar sizing, limit_percent or checkpoint-length logic that needs a length).","commonSituations":"Streaming datasets, TFRecord/JSON-lines iterable datasets, WebDataset-style pipelines inside a combined loader.","solutions":["Use map-style datasets with __len__ where a length is required","Avoid features that need total length (adjust val_check_interval / limits so length isn't queried)","Wrap iterables in a dataset exposing an approximate __len__"],"exampleFix":"# before\ncl = CombinedLoader([DataLoader(IterableDS()), DataLoader(IterableDS())])\nlen(cl._dataset_length())  # NotImplementedError path\n# after\nclass SizedIterable(IterableDS):\n    def __len__(self): return 1000\ncl = CombinedLoader([DataLoader(SizedIterable()), DataLoader(SizedIterable())])","handlingStrategy":"type-guard","validationCode":"from lightning.pytorch.utilities.data import sized_len\ndef has_sized_dataset(cl) -> bool:\n    return any(sized_len(getattr(dl, \"dataset\", None)) is not None for dl in cl.flattened)","typeGuard":"def loader_has_length(dl) -> bool:\n    from lightning.pytorch.utilities.data import sized_len\n    return sized_len(getattr(dl, \"dataset\", None)) is not None","tryCatchPattern":"try:\n    total = cl._dataset_length()\nexcept NotImplementedError:\n    total = None  # streaming mode; skip length-dependent features","preventionTips":["Give iterable datasets an approximate __len__ if length-based features are needed","Design training loops to not require total length for streaming data"],"tags":["combined-loader","iterable-dataset","not-implemented"],"backgroundTag":"iterable-dataset-no-length","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}