Lightning-AI/pytorch-lightning · error · ValueError

Currently only one optimizer is supported with DeepSpeed. Go

Error message

Currently only one optimizer is supported with DeepSpeed. Got {len(optimizers)} optimizers instead.

What it means

DeepSpeed's engine initialization takes exactly one optimizer. DeepSpeedStrategy.setup_module_and_optimizers raises ValueError if the optimizers list has any length other than 1, because the DeepSpeedEngine API cannot represent multiple optimizers.

Source

Thrown at src/lightning/fabric/strategies/deepspeed.py:351

    @property
    def model(self) -> "DeepSpeedEngine":
        return self._deepspeed_engine

    @override
    def setup_module_and_optimizers(
        self, module: Module, optimizers: list[Optimizer], scheduler: Optional["_LRScheduler"] = None
    ) -> tuple["DeepSpeedEngine", list[Optimizer], Any]:
        """Set up a model and multiple optimizers together, along with an optional learning rate scheduler. Currently,
        only a single optimizer is supported.

        Return:
            The model wrapped into a :class:`deepspeed.DeepSpeedEngine`, a list with a single
            deepspeed optimizer, and an optional learning rate scheduler.

        """
        if len(optimizers) != 1:
            raise ValueError(
                f"Currently only one optimizer is supported with DeepSpeed. Got {len(optimizers)} optimizers instead."
            )

        self._deepspeed_engine, optimizer, scheduler = self._initialize_engine(module, optimizers[0], scheduler)
        self._set_deepspeed_activation_checkpointing()
        return self._deepspeed_engine, [optimizer], scheduler

    @override
    def setup_module(self, module: Module) -> "DeepSpeedEngine":
        """Set up a module for inference (no optimizers).

        For training, see :meth:`setup_module_and_optimizers`.

        """
        self._deepspeed_engine, _, _ = self._initialize_engine(module)
        return self._deepspeed_engine

    @override

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Restructure training to a single optimizer (e.g. combine parameters into one optimizer, or train models in separate Fabric runs/checkpoints)
2. If you need multiple models, create one Fabric/DeepSpeedStrategy instance per model
3. Use a different strategy (DDP/FSDP) if multiple optimizers are a hard requirement

Example fix

# before
model, (opt_g, opt_d) = fabric.setup(module, opt_g, opt_d)  # under deepspeed
# after
model, opt_g = fabric.setup(module, opt_g)
# train discriminator in a separate run/stage

assert len(optimizers) == 1, "DeepSpeed supports exactly one optimizer"

def deepspeed_optimizer_count_ok(optimizers) -> bool:
    return len(list(optimizers)) == 1

try:
    model, optimizers, scheduler = strategy.setup_module_and_optimizers(module, optimizers)
except ValueError:
    # split training into per-model runs

When it happens

Trigger: fabric.setup(module, optimizer1, optimizer2) or strategy.setup_module_and_optimizers(module, [opt1, opt2]) with more than one optimizer under DeepSpeedStrategy; also passing zero optimizers.

Common situations: GAN-style or multi-model training with separate optimizers; porting a DDP script that sets up several optimizers to DeepSpeed.

Related errors

• `precision={precision!r})` is not supported in DeepSpeed. `p
• To use the `DeepSpeedStrategy`, you must have DeepSpeed inst
• PyTorch >= 2.6 requires DeepSpeed >= 0.16.0. Detected DeepSp
• The `{type(self).__name__}` does not support setting up the
• `{empty_init=}` is not a valid choice with `DeepSpeedStrateg

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/5e5b6bd346190b97. Report an issue: GitHub.