{"record":{"id":"0989d22e4d724642","repo":"Lightning-AI/pytorch-lightning","slug":"you-have-configured-len-self-optimizers-optimiz","errorCode":null,"errorMessage":"You have configured {len(self.optimizers)} optimizers but the checkpoint contains {len(optimizer_states)} optimizers to load. Please resume training with the same number of optimizers or edit the checkpoint manually to remove states.","messagePattern":"You have configured (.+?) optimizers but the checkpoint contains (.+?) optimizers to load\\. Please resume training with the same number of optimizers or edit the checkpoint manually to remove states\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"src/lightning/pytorch/strategies/fsdp.py","lineNumber":671,"sourceCode":"                checkpoint.pop(\"state_dict\"),\n                module=self.model,\n                world_size=self.world_size,\n                strict=self.lightning_module.strict_loading,\n            )\n\n            # Materialize lazy tensors if there are any left in the checkpoint\n            # The `torch.Optimizer.load_state_dict` method can't load lazy tensors because of deepcopy pickle issues\n            checkpoint = _materialize_tensors(checkpoint)\n\n            from torch.distributed.fsdp import FullyShardedDataParallel as FSDP\n            from torch.distributed.fsdp import OptimStateKeyType\n\n            optimizer_states = checkpoint.get(\"optimizer_states\")\n            if optimizer_states is None or self.lightning_module.trainer.state.fn != TrainerFn.FITTING:\n                # If the optimizer states are not present, we don't need to do anything (backward compatibility)\n                return checkpoint\n            if len(self.optimizers) != len(optimizer_states):\n                raise RuntimeError(\n                    f\"You have configured {len(self.optimizers)} optimizers but the checkpoint contains\"\n                    f\" {len(optimizer_states)} optimizers to load. Please resume training with the same number\"\n                    \" of optimizers or edit the checkpoint manually to remove states.\"\n                )\n\n            # rank0_only should be false because we need to load the optimizer state on all ranks\n            with _get_full_state_dict_context(self.model, world_size=self.world_size, rank0_only=False):\n                for optimizer, opt_state in zip(self.optimizers, optimizer_states):\n                    if isinstance(list(opt_state[\"state\"].keys())[0], int):\n                        # Handling the case where the optimizer state is saved from a normal optimizer\n                        opt_state = FSDP.rekey_optim_state_dict(opt_state, OptimStateKeyType.PARAM_NAME, self.model)\n\n                    opt_state = FSDP.optim_state_dict_to_load(\n                        optim_state_dict=opt_state,\n                        model=self.model,\n                        optim=optimizer,\n                    )\n                    optimizer.load_state_dict(opt_state)","sourceCodeStart":653,"sourceCodeEnd":689,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/pytorch/strategies/fsdp.py#L653-L689","documentation":"On resume, FSDP's load_checkpoint must redistribute optimizer states to the flattened parameters, which only works if the number of optimizers in the current run matches the number of optimizer state entries in the checkpoint. A mismatch raises RuntimeError asking you to align counts or edit the checkpoint.","triggerScenarios":"`trainer.fit(model, ckpt_path=...)` under FSDPStrategy where `configure_optimizers` now returns N optimizers but the checkpoint saved M != N optimizer_states (e.g. checkpoint from a multi-optimizer run resumed with one optimizer, or vice versa).","commonSituations":"Refactoring a model from two optimizers to one between runs and resuming from an old checkpoint; resuming a checkpoint saved by a different module/strategy; fine-tuning scripts that drop an optimizer but reuse ckpt paths.","solutions":["Make `configure_optimizers` return the same number of optimizers as saved in the checkpoint","Or start fresh from just the weights: load with `trainer.strategy.load_checkpoint` on a weights-only checkpoint / re-save a weights-only checkpoint","Or programmatically strip extra `optimizer_states` entries from the checkpoint dict and re-save it"],"exampleFix":"# before\n# checkpoint has 2 optimizer_states, current model returns 1\ndef configure_optimizers(self):\n    return torch.optim.AdamW(self.parameters())\n\n# after\nckpt = torch.load(\"last.ckpt\", map_location=\"cpu\")\nckpt[\"optimizer_states\"] = ckpt[\"optimizer_states\"][:1]\ntorch.save(ckpt, \"last_trimmed.ckpt\")\ntrainer.fit(model, ckpt_path=\"last_trimmed.ckpt\")","handlingStrategy":"try-catch","validationCode":"ckpt = torch.load(ckpt_path, map_location=\"cpu\")\nsaved = len(ckpt.get(\"optimizer_states\", []))\nconfigured = 1  # number your configure_optimizers returns\nif saved != configured:\n    ckpt[\"optimizer_states\"] = ckpt[\"optimizer_states\"][:configured]\n    torch.save(ckpt, trimmed_path)","typeGuard":null,"tryCatchPattern":"try:\n    trainer.fit(model, ckpt_path=ckpt)\nexcept RuntimeError as e:\n    if \"optimizers to load\" in str(e):\n        # trim optimizer_states in the checkpoint or align configure_optimizers, then retry\n        ...\n    raise","preventionTips":["Keep the optimizer count stable across resumes of the same experiment","When changing optimizer topology, save/load weights-only checkpoints","Sanity-check optimizer_states length in the checkpoint before trainer.fit"],"tags":["fsdp","resume","optimizer-states","checkpoint-mismatch"],"backgroundTag":"optimizer-count-checkpoint-mismatch","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}