{"record":{"id":"7b7f7d51f95240d2","repo":"Lightning-AI/pytorch-lightning","slug":"when-using-an-iterabledataset-for-train-dataloade","errorCode":null,"errorMessage":"When using an IterableDataset for `train_dataloader`, `Trainer(val_check_interval)` must be time based, `1.0` or an int. An int k specifies checking validation every k training batches.","messagePattern":"When using an IterableDataset for `train_dataloader`, `Trainer\\(val_check_interval\\)` must be time based, `1\\.0` or an int\\. An int k specifies checking validation every k training batches\\.","errorType":"exception","errorClass":"MisconfigurationException","httpStatus":null,"severity":"error","filePath":"src/lightning/pytorch/loops/fit_loop.py","lineNumber":311,"sourceCode":"        elif isinstance(trainer.val_check_interval, int):\n            trainer.val_check_batch = trainer.val_check_interval\n            if (\n                trainer.val_check_batch > self.max_batches\n                and trainer.check_val_every_n_epoch is not None\n                and trainer.limit_val_batches > 0\n            ):\n                raise ValueError(\n                    f\" `val_check_interval` ({trainer.val_check_interval}) must be less than or equal\"\n                    f\" to the number of the training batches ({self.max_batches}).\"\n                    \" If you want to disable validation set `limit_val_batches` to 0.0 instead.\"\n                    \" If you want to validate based on the total training batches, set `check_val_every_n_epoch=None`.\"\n                )\n        else:\n            if not has_len_all_ranks_:\n                if trainer.val_check_interval == 1.0:\n                    trainer.val_check_batch = float(\"inf\")\n                else:\n                    raise MisconfigurationException(\n                        \"When using an IterableDataset for `train_dataloader`,\"\n                        \" `Trainer(val_check_interval)` must be time based, `1.0` or an int. An int k specifies\"\n                        \" checking validation every k training batches.\"\n                    )\n            else:\n                trainer.val_check_batch = int(self.max_batches * trainer.val_check_interval)\n                trainer.val_check_batch = max(1, trainer.val_check_batch)\n\n        if trainer.loggers and self.max_batches < trainer.log_every_n_steps and not trainer.fast_dev_run:\n            rank_zero_warn(\n                f\"The number of training batches ({self.max_batches}) is smaller than the logging interval\"\n                f\" Trainer(log_every_n_steps={trainer.log_every_n_steps}). Set a lower value for log_every_n_steps if\"\n                \" you want to see logs for the training epoch.\",\n                category=PossibleUserWarning,\n            )\n\n        if self.max_batches < trainer.accumulate_grad_batches:\n            rank_zero_warn(","sourceCodeStart":293,"sourceCodeEnd":329,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/pytorch/loops/fit_loop.py#L293-L329","documentation":"Raised in FitLoop.setup_data when the train dataloader has no length on all ranks (typical for IterableDataset) and val_check_interval is neither time-based nor exactly 1.0 nor an int. With unlengthed iterable data Lightning cannot map a fractional interval to a batch count, so only 1.0 (validate every epoch, encoded as inf) or integer batch counts are accepted.","triggerScenarios":"Using an IterableDataset for train_dataloader together with `Trainer(val_check_interval=0.5)` or any fractional value other than 1.0; streaming datasets (TFRecords, Kafka, webdataset streams) that do not implement __len__.","commonSituations":"Migrating a map-style dataset pipeline (with val_check_interval=0.25) to a streaming pipeline; reusing Trainer flags from a standard DataLoader setup with an iterable-style dataset; large-scale data pipelines using StreamingDataset.","solutions":["Set val_check_interval=1.0 to validate once per epoch","Use an integer val_check_interval (e.g. 500) to validate every k batches","Use a time-based interval via `Trainer(val_check_interval=<timedelta>)` with a TQDMProgressBar with refresh_rate","Wrap the iterable dataset in a length-aware dataset if a fraction is essential"],"exampleFix":"# before\ntrainer = pl.Trainer(val_check_interval=0.5)  # train_dataloader is an IterableDataset\n\n# after\ntrainer = pl.Trainer(val_check_interval=500)  # validate every 500 batches","handlingStrategy":"validation","validationCode":"from torch.utils.data import IterableDataset\n\nis_iterable = isinstance(train_dataloader.dataset, IterableDataset) or len(hasattr(train_dataloader, '__len__') and train_dataloader) == 0\nif is_iterable and isinstance(trainer_config['val_check_interval'], float) and trainer_config['val_check_interval'] != 1.0:\n    trainer_config['val_check_interval'] = 1.0  # or int k / timedelta\ntrainer = pl.Trainer(**trainer_config)","typeGuard":"def vci_supported_for_iterable(vci) -> bool:\n    if isinstance(vci, float):\n        return vci == 1.0\n    return isinstance(vci, int) or isinstance(vci, timedelta)","tryCatchPattern":null,"preventionTips":["When using IterableDataset, always set val_check_interval to 1.0, an int, or a timedelta","Centralize Trainer flags per data-regime (map-style vs iterable) in config profiles"],"tags":["pytorch-lightning","iterable-dataset","streaming","validation"],"backgroundTag":"iterable-dataset-length-unsupported","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}