Lightning-AI/pytorch-lightning · critical · RuntimeError

The CombinedLoader has {len(stateful_loaders)} stateful load

Error message

The CombinedLoader has {len(stateful_loaders)} stateful loaders, but found {len(states)} states in the checkpoint. Please make sure you define the same dataloaders that were used when saving the checkpoint.

What it means

When restoring, CombinedLoader._load_state_dicts matches checkpointed states to stateful loaders (those implementing _Stateful/load_state_dict). If the count of stateful loaders differs from the number of states saved in the checkpoint, it raises RuntimeError because loader-to-state pairing is ambiguous.

Source

Thrown at src/lightning/pytorch/utilities/combined_loader.py:388

        """Compute the total length of the datasets according to the current mode."""
        datasets = [getattr(dl, "dataset", None) for dl in self.flattened]
        lengths = [length for ds in datasets if (length := sized_len(ds)) is not None]
        if not lengths:
            raise NotImplementedError("All datasets are iterable-style datasets.")
        fn = _SUPPORTED_MODES[self._mode]["fn"]
        return fn(lengths)

    def _state_dicts(self) -> list[dict[str, Any]]:
        """Returns the list of state dicts for iterables in `self.flattened` that are stateful."""
        return [loader.state_dict() for loader in self.flattened if isinstance(loader, _Stateful)]

    def _load_state_dicts(self, states: list[dict[str, Any]]) -> None:
        """Loads the state dicts for iterables in `self.flattened` that are stateful."""
        if not states:
            return
        stateful_loaders = [loader for loader in self.flattened if isinstance(loader, _Stateful)]
        if len(stateful_loaders) != len(states):
            raise RuntimeError(
                f"The CombinedLoader has {len(stateful_loaders)} stateful loaders, but found {len(states)} states"
                " in the checkpoint. Please make sure you define the same dataloaders that were used when saving"
                " the checkpoint."
            )
        for loader, state_dict in zip(stateful_loaders, states):
            loader.load_state_dict(state_dict)


def _shutdown_workers_and_reset_iterator(dataloader: object) -> None:
    if hasattr(dataloader, "_iterator"):
        if isinstance(dataloader._iterator, _MultiProcessingDataLoaderIter):
            del dataloader._iterator
        dataloader._iterator = None


def _get_iterables_lengths(iterables: list[Iterable]) -> list[Union[int, float]]:
    return [(float("inf") if (length := sized_len(iterable)) is None else length) for iterable in iterables]

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Define exactly the same stateful dataloaders (same count) as when the checkpoint was saved
2. If the run config changed intentionally, start fresh without ckpt_path or save a new checkpoint
3. Verify you import LightningModule/dataloaders from the same lightning.pytorch namespace as the checkpoint run

Example fix

# before
# checkpoint saved with 2 stateful loaders
cl = CombinedLoader([stateful_dl])  # now only 1
trainer.fit(model, ckpt_path="old.ckpt")
# after
cl = CombinedLoader([stateful_dl1, stateful_dl2])  # match saved topology
trainer.fit(model, ckpt_path="old.ckpt")

from lightning.pytorch.utilitiescombined_loader import _Stateful  # note actual import path
n_stateful = sum(isinstance(l, _Stateful) for l in cl.flattened)
# compare n_stateful against len(states) from the checkpoint before loading

try:
    cl._load_state_dicts(states)
except RuntimeError as e:
    if "stateful loaders" in str(e):
        raise SystemExit("Dataloader topology changed; cannot resume this checkpoint") from e
    raise

When it happens

Trigger: Resuming from a checkpoint saved with a different number of stateful dataloaders (e.g. 2 IterableDatasets with state before, 1 now), via _load_combined_loader_states during trainer.fit(ckpt_path=...).

Common situations: Changing dataloader topology between runs; resuming an old checkpoint after refactoring combined loaders; mixed import paths causing a loader to not register as stateful.

Related errors

• `.{fn}(ckpt_path="hpc")` is set but no HPC checkpoint was fo
• `.{fn}()` found no path for the best weights: {ckpt_path!r}.
• Trying to restore optimizer state but checkpoint contains on
• Trying to restore learning rate scheduler state but checkpoi
• `Trainer.save_checkpoint(..., storage_options=...)` with `st

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/cdd19505349033bc. Report an issue: GitHub.