Lightning-AI/pytorch-lightning · error · MisconfigurationException

SWA currently not supported for more than 1 `lr_scheduler`.

Error message

SWA currently not supported for more than 1 `lr_scheduler`.

What it means

StochasticWeightAveraging supports at most one learning-rate scheduler (it replaces scheduling with a constant/annealed SWA LR after swa_epoch_start). At on_fit_start, more than one lr_scheduler config triggers this MisconfigurationException.

Source

Thrown at src/lightning/pytorch/callbacks/stochastic_weight_avg.py:164

    @staticmethod
    def pl_module_contains_batch_norm(pl_module: "pl.LightningModule") -> bool:
        return any(isinstance(module, nn.modules.batchnorm._BatchNorm) for module in pl_module.modules())

    @override
    def setup(self, trainer: "pl.Trainer", pl_module: "pl.LightningModule", stage: str) -> None:
        if isinstance(trainer.strategy, (FSDPStrategy, DeepSpeedStrategy)):
            raise MisconfigurationException("SWA does not currently support sharded models.")

        # copy the model before moving it to accelerator device.
        self._average_model = deepcopy(pl_module)

    @override
    def on_fit_start(self, trainer: "pl.Trainer", pl_module: "pl.LightningModule") -> None:
        if len(trainer.optimizers) != 1:
            raise MisconfigurationException("SWA currently works with 1 `optimizer`.")

        if len(trainer.lr_scheduler_configs) > 1:
            raise MisconfigurationException("SWA currently not supported for more than 1 `lr_scheduler`.")

        assert trainer.max_epochs is not None
        if isinstance(self._swa_epoch_start, float):
            if trainer.max_epochs == -1:
                raise MisconfigurationException(
                    "SWA with `swa_epoch_start` as a float is not supported when `max_epochs=-1`. "
                    "Please provide `swa_epoch_start` as an integer."
                )
            self._swa_epoch_start = int(trainer.max_epochs * self._swa_epoch_start)

        self._model_contains_batch_norm = self.pl_module_contains_batch_norm(pl_module)

        self._max_epochs = trainer.max_epochs
        if self._model_contains_batch_norm and trainer.max_epochs != -1:
            # virtually increase max_epochs to perform batch norm update on latest epoch.
            assert trainer.fit_loop.max_epochs is not None
            trainer.fit_loop.max_epochs += 1

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Merge schedulers into one (e.g. use SequentialLR or a single cosine schedule with warmup)
2. Or keep only the primary scheduler
3. Or remove the SWA callback if multiple schedulers are required

Example fix

# before
def configure_optimizers(self):
    opt = torch.optim.AdamW(self.parameters())
    return [opt], [torch.optim.lr_scheduler.StepLR(opt, 10), torch.optim.lr_scheduler.ReduceLROnPlateau(opt)]
# after
from torch.optim.lr_scheduler import SequentialLR
opt = torch.optim.AdamW(self.parameters())
sched = SequentialLR(opt, [WarmupLR(opt, 5), CosineAnnealingLR(opt, 95)], milestones=[5])
return [opt], [sched]

from torch.optim.lr_scheduler import SequentialLR
def configure_optimizers(self):
    opt = torch.optim.AdamW(self.parameters(), lr=1e-3)
    sched = SequentialLR(opt, self.sched_list, milestones=self.milestones)
    return [opt], [sched]  # exactly one scheduler config

When it happens

Trigger: configure_optimizers returns one optimizer with a list of 2+ LRScheduler/LightningModule hyperparameter dict scheduler entries, e.g. [ReduceLROnPlateau, StepLR], with SWA enabled.

Common situations: Combining warmup + decay as two separate schedulers instead of chained/sequential schedulers; adding SWA to an existing multi-scheduler setup.

Related errors

• swa_epoch_start should be a >0 integer or a float between 0
• The `avg_fn` should be callable.
• device is expected to be a torch.device or a str. Found {dev
• SWA currently works with 1 `optimizer`.
• SWA with `swa_epoch_start` as a float is not supported when

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/b7b1e5da1886ee2a. Report an issue: GitHub.