Lightning-AI/pytorch-lightning · error · MisconfigurationException

SWA currently works with 1 `optimizer`.

Error message

SWA currently works with 1 `optimizer`.

What it means

At on_fit_start, StochasticWeightAveraging verifies the trainer has exactly one optimizer because its averaging and LR-constant logic only handles a single optimizer. If configure_optimizers returned more than one, this MisconfigurationException is raised.

Source

Thrown at src/lightning/pytorch/callbacks/stochastic_weight_avg.py:161

            return float("inf")  # type: ignore[return-value]
        return self._max_epochs - 1  # 0-based

    @staticmethod
    def pl_module_contains_batch_norm(pl_module: "pl.LightningModule") -> bool:
        return any(isinstance(module, nn.modules.batchnorm._BatchNorm) for module in pl_module.modules())

    @override
    def setup(self, trainer: "pl.Trainer", pl_module: "pl.LightningModule", stage: str) -> None:
        if isinstance(trainer.strategy, (FSDPStrategy, DeepSpeedStrategy)):
            raise MisconfigurationException("SWA does not currently support sharded models.")

        # copy the model before moving it to accelerator device.
        self._average_model = deepcopy(pl_module)

    @override
    def on_fit_start(self, trainer: "pl.Trainer", pl_module: "pl.LightningModule") -> None:
        if len(trainer.optimizers) != 1:
            raise MisconfigurationException("SWA currently works with 1 `optimizer`.")

        if len(trainer.lr_scheduler_configs) > 1:
            raise MisconfigurationException("SWA currently not supported for more than 1 `lr_scheduler`.")

        assert trainer.max_epochs is not None
        if isinstance(self._swa_epoch_start, float):
            if trainer.max_epochs == -1:
                raise MisconfigurationException(
                    "SWA with `swa_epoch_start` as a float is not supported when `max_epochs=-1`. "
                    "Please provide `swa_epoch_start` as an integer."
                )
            self._swa_epoch_start = int(trainer.max_epochs * self._swa_epoch_start)

        self._model_contains_batch_norm = self.pl_module_contains_batch_norm(pl_module)

        self._max_epochs = trainer.max_epochs
        if self._model_contains_batch_norm and trainer.max_epochs != -1:
            # virtually increase max_epochs to perform batch norm update on latest epoch.

View on GitHub (pinned to 9fed5c27d2)

Solutions

  1. Consolidate into one optimizer with multiple param groups: torch.optim.Adam([{ 'params': a.parameters()}, {'params': b.parameters(), 'lr': 1e-4}])
  2. Or remove SWA when multiple optimizers are genuinely required

Example fix

# before
def configure_optimizers(self):
    return [torch.optim.Adam(self.enc.parameters()), torch.optim.Adam(self.dec.parameters())]
# after
def configure_optimizers(self):
    opt = torch.optim.Adam([
        {"params": self.enc.parameters()},
        {"params": self.dec.parameters(), "lr": 1e-4},
    ])
    return opt
Defensive patterns

Strategy: validation

Validate before calling

class MyModule(LightningModule):
    def configure_optimizers(self):
        if getattr(self, '_n_optimizers', 1) > 1 and self.use_swa:
            raise ValueError('SWA requires one optimizer; merge param groups')
        return torch.optim.Adam([
            {"params": self.enc.parameters()},
            {"params": self.dec.parameters(), "lr": 1e-4},
        ])

Prevention

When it happens

Trigger: configure_optimizers returns a list/tuple of 2+ optimizers (multiple models or param groups per optimizer is fine, multiple optimizer objects is not) while SWA is in callbacks.

Common situations: Using separate optimizers for generator/discriminator, or per-module optimizers, and adding SWA for better checkpoints.

Related errors

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/76e04e686198009a. Report an issue: GitHub.