Lightning-AI/pytorch-lightning · error · MisconfigurationException

f"When using an `IterableDataset`, `Trainer(limit_{stage.dat

Error message

f"When using an `IterableDataset`, `Trainer(limit_{stage.dataloader_prefix}_batches)` must be" f" `1.0` or an int. An int specifies `num_{stage.dataloader_prefix}_batches` to use."

What it means

With IterableDatasets the dataset length is unknown, so Lightning cannot take a fractional percentage of batches. _parse_num_batches requires limit_{train,val,test,predict}_batches to be either 1.0 or an integer when the dataloader length is infinite (IterableDataset without __len__). Any other float raises this MisconfigurationException.

Source

Thrown at src/lightning/pytorch/trainer/connectors/data_connector.py:462

            " (https://github.com/pytorch/pytorch/issues/91252). We recommend setting `pin_memory=False` in this case.",
            category=PossibleUserWarning,
        )


def _parse_num_batches(
    stage: RunningStage, length: Union[int, float], limit_batches: Union[int, float]
) -> Union[int, float]:
    if length == 0:
        return int(length)

    num_batches = length
    # limit num batches either as a percent or num steps
    if isinstance(limit_batches, int):
        num_batches = min(length, limit_batches)
    elif isinstance(limit_batches, float) and length != float("inf"):
        num_batches = int(length * limit_batches)
    elif limit_batches != 1.0:
        raise MisconfigurationException(
            f"When using an `IterableDataset`, `Trainer(limit_{stage.dataloader_prefix}_batches)` must be"
            f" `1.0` or an int. An int specifies `num_{stage.dataloader_prefix}_batches` to use."
        )

    if num_batches == 0 and limit_batches > 0.0 and isinstance(limit_batches, float) and length != float("inf"):
        min_percentage = 1.0 / length
        raise MisconfigurationException(
            f"You requested to check {limit_batches} of the `{stage.dataloader_prefix}_dataloader` but"
            f" {limit_batches} * {length} < 1. Please increase the"
            f" `limit_{stage.dataloader_prefix}_batches` argument. Try at least"
            f" `limit_{stage.dataloader_prefix}_batches={min_percentage}`"
        )
    return num_batches


def _process_dataloader(
    trainer: "pl.Trainer", trainer_fn: TrainerFn, stage: RunningStage, dataloader: object
) -> object:

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Set the flag to 1.0 or an int: Trainer(limit_train_batches=1.0) or Trainer(limit_train_batches=100)
2. Use limit_batches=0.0 only via the int 0 (Trainer(limit_train_batches=0)) to skip the stage, since 0 is an int
3. Give the IterableDataset a __len__ so length is not inf and percentages become valid

Example fix

# before
trainer = Trainer(limit_train_batches=0.5)  # with IterableDataset

# after
trainer = Trainer(limit_train_batches=1.0)
# or
trainer = Trainer(limit_train_batches=100)

from torch.utils.data import IterableDataset

def uses_iterable(dl) -> bool:
    ds = getattr(dl, "dataset", None)
    return isinstance(ds, IterableDataset) or (hasattr(dl, "__len__") is False)

if uses_iterable(model.train_dataloader()):
    assert limit_train_batches == 1.0 or isinstance(limit_train_batches, int)

from typing import Union

def is_safe_limit(value: Union[float, int], has_iterable: bool) -> bool:
    if not has_iterable:
        return True
    return value == 1.0 or isinstance(value, int)

When it happens

Trigger: Trainer(limit_train_batches=0.5) (or limit_val_batches=0.25, etc.) combined with a dataloader whose len is float('inf'), i.e. an IterableDataset without __len__; also with combined dataloaders where one uses IterableDataset. Raised during data setup (setup_data) before training begins.

Common situations: Copying a Trainer config from a map-style dataset workflow to a streaming/IterableDataset workflow; using limit_val_batches=0.0 with iterable data (0.0 is not 1.0 and not an int); streaming data with WebDataset or Kafka-style sources.

Related errors

• f"You requested to check {limit_batches} of the `{stage.data
• Device should be CPU, got {device} instead.
• You requested to find {num_devices} devices but there are no
• You are trying to `self.log()` but the loop's result collect
• You are trying to `self.log()` but it is not managed by the

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/6ca47c4c7a8c2082. Report an issue: GitHub.