{"record":{"id":"6ca47c4c7a8c2082","repo":"Lightning-AI/pytorch-lightning","slug":"f-when-using-an-iterabledataset-trainer-limit","errorCode":null,"errorMessage":"f\"When using an `IterableDataset`, `Trainer(limit_{stage.dataloader_prefix}_batches)` must be\" f\" `1.0` or an int. An int specifies `num_{stage.dataloader_prefix}_batches` to use.\"","messagePattern":"f\"When using an `IterableDataset`, `Trainer\\(limit_(.+?)_batches\\)` must be\" f\" `1\\.0` or an int\\. An int specifies `num_(.+?)_batches` to use\\.\"","errorType":"validation","errorClass":"MisconfigurationException","httpStatus":null,"severity":"error","filePath":"src/lightning/pytorch/trainer/connectors/data_connector.py","lineNumber":462,"sourceCode":"            \" (https://github.com/pytorch/pytorch/issues/91252). We recommend setting `pin_memory=False` in this case.\",\n            category=PossibleUserWarning,\n        )\n\n\ndef _parse_num_batches(\n    stage: RunningStage, length: Union[int, float], limit_batches: Union[int, float]\n) -> Union[int, float]:\n    if length == 0:\n        return int(length)\n\n    num_batches = length\n    # limit num batches either as a percent or num steps\n    if isinstance(limit_batches, int):\n        num_batches = min(length, limit_batches)\n    elif isinstance(limit_batches, float) and length != float(\"inf\"):\n        num_batches = int(length * limit_batches)\n    elif limit_batches != 1.0:\n        raise MisconfigurationException(\n            f\"When using an `IterableDataset`, `Trainer(limit_{stage.dataloader_prefix}_batches)` must be\"\n            f\" `1.0` or an int. An int specifies `num_{stage.dataloader_prefix}_batches` to use.\"\n        )\n\n    if num_batches == 0 and limit_batches > 0.0 and isinstance(limit_batches, float) and length != float(\"inf\"):\n        min_percentage = 1.0 / length\n        raise MisconfigurationException(\n            f\"You requested to check {limit_batches} of the `{stage.dataloader_prefix}_dataloader` but\"\n            f\" {limit_batches} * {length} < 1. Please increase the\"\n            f\" `limit_{stage.dataloader_prefix}_batches` argument. Try at least\"\n            f\" `limit_{stage.dataloader_prefix}_batches={min_percentage}`\"\n        )\n    return num_batches\n\n\ndef _process_dataloader(\n    trainer: \"pl.Trainer\", trainer_fn: TrainerFn, stage: RunningStage, dataloader: object\n) -> object:","sourceCodeStart":444,"sourceCodeEnd":480,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/pytorch/trainer/connectors/data_connector.py#L444-L480","documentation":"With IterableDatasets the dataset length is unknown, so Lightning cannot take a fractional percentage of batches. _parse_num_batches requires limit_{train,val,test,predict}_batches to be either 1.0 or an integer when the dataloader length is infinite (IterableDataset without __len__). Any other float raises this MisconfigurationException.","triggerScenarios":"Trainer(limit_train_batches=0.5) (or limit_val_batches=0.25, etc.) combined with a dataloader whose len is float('inf'), i.e. an IterableDataset without __len__; also with combined dataloaders where one uses IterableDataset. Raised during data setup (setup_data) before training begins.","commonSituations":"Copying a Trainer config from a map-style dataset workflow to a streaming/IterableDataset workflow; using limit_val_batches=0.0 with iterable data (0.0 is not 1.0 and not an int); streaming data with WebDataset or Kafka-style sources.","solutions":["Set the flag to 1.0 or an int: Trainer(limit_train_batches=1.0) or Trainer(limit_train_batches=100)","Use limit_batches=0.0 only via the int 0 (Trainer(limit_train_batches=0)) to skip the stage, since 0 is an int","Give the IterableDataset a __len__ so length is not inf and percentages become valid"],"exampleFix":"# before\ntrainer = Trainer(limit_train_batches=0.5)  # with IterableDataset\n\n# after\ntrainer = Trainer(limit_train_batches=1.0)\n# or\ntrainer = Trainer(limit_train_batches=100)","handlingStrategy":"validation","validationCode":"from torch.utils.data import IterableDataset\n\ndef uses_iterable(dl) -> bool:\n    ds = getattr(dl, \"dataset\", None)\n    return isinstance(ds, IterableDataset) or (hasattr(dl, \"__len__\") is False)\n\nif uses_iterable(model.train_dataloader()):\n    assert limit_train_batches == 1.0 or isinstance(limit_train_batches, int)","typeGuard":"from typing import Union\n\ndef is_safe_limit(value: Union[float, int], has_iterable: bool) -> bool:\n    if not has_iterable:\n        return True\n    return value == 1.0 or isinstance(value, int)","tryCatchPattern":null,"preventionTips":["Default to integer limits (limit_train_batches=100) — they work in both regimes","Avoid fractional limits entirely when streaming data","Document which dataloaders are IterableDataset in the team config"],"tags":["pytorch-lightning","iterable-dataset","limit-batches","misconfiguration"],"backgroundTag":"fractional-limit-on-unknown-length","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}