Lightning-AI/pytorch-lightning · critical · RuntimeError

You have configured {len(self.optimizers)} optimizers but th

Error message

You have configured {len(self.optimizers)} optimizers but the checkpoint contains {len(optimizer_states)} optimizers to load. Please resume training with the same number of optimizers or edit the checkpoint manually to remove states.

What it means

On resume, FSDP's load_checkpoint must redistribute optimizer states to the flattened parameters, which only works if the number of optimizers in the current run matches the number of optimizer state entries in the checkpoint. A mismatch raises RuntimeError asking you to align counts or edit the checkpoint.

Source

Thrown at src/lightning/pytorch/strategies/fsdp.py:671

                checkpoint.pop("state_dict"),
                module=self.model,
                world_size=self.world_size,
                strict=self.lightning_module.strict_loading,
            )

            # Materialize lazy tensors if there are any left in the checkpoint
            # The `torch.Optimizer.load_state_dict` method can't load lazy tensors because of deepcopy pickle issues
            checkpoint = _materialize_tensors(checkpoint)

            from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
            from torch.distributed.fsdp import OptimStateKeyType

            optimizer_states = checkpoint.get("optimizer_states")
            if optimizer_states is None or self.lightning_module.trainer.state.fn != TrainerFn.FITTING:
                # If the optimizer states are not present, we don't need to do anything (backward compatibility)
                return checkpoint
            if len(self.optimizers) != len(optimizer_states):
                raise RuntimeError(
                    f"You have configured {len(self.optimizers)} optimizers but the checkpoint contains"
                    f" {len(optimizer_states)} optimizers to load. Please resume training with the same number"
                    " of optimizers or edit the checkpoint manually to remove states."
                )

            # rank0_only should be false because we need to load the optimizer state on all ranks
            with _get_full_state_dict_context(self.model, world_size=self.world_size, rank0_only=False):
                for optimizer, opt_state in zip(self.optimizers, optimizer_states):
                    if isinstance(list(opt_state["state"].keys())[0], int):
                        # Handling the case where the optimizer state is saved from a normal optimizer
                        opt_state = FSDP.rekey_optim_state_dict(opt_state, OptimStateKeyType.PARAM_NAME, self.model)

                    opt_state = FSDP.optim_state_dict_to_load(
                        optim_state_dict=opt_state,
                        model=self.model,
                        optim=optimizer,
                    )
                    optimizer.load_state_dict(opt_state)

View on GitHub (pinned to 9fed5c27d2)

Solutions

  1. Make `configure_optimizers` return the same number of optimizers as saved in the checkpoint
  2. Or start fresh from just the weights: load with `trainer.strategy.load_checkpoint` on a weights-only checkpoint / re-save a weights-only checkpoint
  3. Or programmatically strip extra `optimizer_states` entries from the checkpoint dict and re-save it

Example fix

# before
# checkpoint has 2 optimizer_states, current model returns 1
def configure_optimizers(self):
    return torch.optim.AdamW(self.parameters())

# after
ckpt = torch.load("last.ckpt", map_location="cpu")
ckpt["optimizer_states"] = ckpt["optimizer_states"][:1]
torch.save(ckpt, "last_trimmed.ckpt")
trainer.fit(model, ckpt_path="last_trimmed.ckpt")
Defensive patterns

Strategy: try-catch

Validate before calling

ckpt = torch.load(ckpt_path, map_location="cpu")
saved = len(ckpt.get("optimizer_states", []))
configured = 1  # number your configure_optimizers returns
if saved != configured:
    ckpt["optimizer_states"] = ckpt["optimizer_states"][:configured]
    torch.save(ckpt, trimmed_path)

Try / catch

try:
    trainer.fit(model, ckpt_path=ckpt)
except RuntimeError as e:
    if "optimizers to load" in str(e):
        # trim optimizer_states in the checkpoint or align configure_optimizers, then retry
        ...
    raise

Prevention

When it happens

Trigger: `trainer.fit(model, ckpt_path=...)` under FSDPStrategy where `configure_optimizers` now returns N optimizers but the checkpoint saved M != N optimizer_states (e.g. checkpoint from a multi-optimizer run resumed with one optimizer, or vice versa).

Common situations: Refactoring a model from two optimizers to one between runs and resuming from an old checkpoint; resuming a checkpoint saved by a different module/strategy; fine-tuning scripts that drop an optimizer but reuse ckpt paths.

Related errors

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/0989d22e4d724642. Report an issue: GitHub.