Lightning-AI/pytorch-lightning · error · ValueError

Currently only one optimizer is supported with DeepSpeed. Go

Error message

Currently only one optimizer is supported with DeepSpeed. Got {len(optimizers)} optimizers instead.

What it means

DeepSpeed (as integrated by Lightning) wraps exactly one optimizer into its engine, so _setup_model_and_optimizers rejects a list whose length differs from 1. `configure_optimizers` returning multiple optimizers is therefore unsupported under this strategy.

Source

Thrown at src/lightning/pytorch/strategies/deepspeed.py:420

    @override
    def restore_checkpoint_after_setup(self) -> bool:
        return True

    @override
    def _setup_model_and_optimizers(
        self, model: Module, optimizers: list[Optimizer]
    ) -> tuple["deepspeed.DeepSpeedEngine", list[Optimizer]]:
        """Setup a model and multiple optimizers together.

        Currently only a single optimizer is supported.

        Return:
            The model wrapped into a :class:`deepspeed.DeepSpeedEngine` and a list with a single
            deepspeed optimizer.

        """
        if len(optimizers) != 1:
            raise ValueError(
                f"Currently only one optimizer is supported with DeepSpeed. Got {len(optimizers)} optimizers instead."
            )

        # train_micro_batch_size_per_gpu is used for throughput logging purposes
        # normally we set this to the batch size, but it is not available here unless the user provides it
        # as part of the config
        assert self.config is not None
        self.config.setdefault("train_micro_batch_size_per_gpu", 1)
        self.model, optimizer = self._setup_model_and_optimizer(model, optimizers[0])
        self._set_deepspeed_activation_checkpointing()
        return self.model, [optimizer]

    def _setup_model_and_optimizer(
        self,
        model: Module,
        optimizer: Optional[Optimizer],
        lr_scheduler: Optional[Union[LRScheduler, ReduceLROnPlateau]] = None,
    ) -> tuple["deepspeed.DeepSpeedEngine", Optimizer]:

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Refactor `configure_optimizers` to return a single optimizer (e.g. use one optimizer over all parameters with param groups)
2. If multiple optimizers are essential, use a different strategy such as DDP
3. Check for accidental list-wrapping, e.g. `return [opt1, opt2]` where one optimizer suffices

Example fix

# before
def configure_optimizers(self):
    return [torch.optim.AdamW(self.g.parameters()), torch.optim.AdamW(self.d.parameters())]

# after
def configure_optimizers(self):
    return torch.optim.AdamW(itertools.chain(self.g.parameters(), self.d.parameters()))

n = len(self.lr_scheduler_configs) if hasattr(self, "lr_scheduler_configs") else 0
optimizers = self.configure_optimizers()  # in tests, assert shape before training
assert not isinstance(optimizers, (list, tuple)) or len(optimizers) == 1, "DeepSpeed supports one optimizer"

When it happens

Trigger: A LightningModule whose `configure_optimizers()` returns a list/tuple of 2+ optimizers (or a dict with multiple optimizers) while training with `DeepSpeedStrategy`.

Common situations: Reusing a GAN / multi-model LightningModule (separate optimizers for generator and discriminator) with DeepSpeed; migrating a manual training loop that stepped several optimizers.

Related errors

• Currently only one optimizer is supported with DeepSpeed. Go
• SWA currently works with 1 `optimizer`.
• `{self.__class__.__name__}.add_configure_optimizers_method_t
• DeepSpeed currently only supports single optimizer, single o
• `model.configure_optimizers()` returned {len(optimizers)}, b

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/7861b598d6aa964c. Report an issue: GitHub.