Lightning-AI/pytorch-lightning · error · MisconfigurationException

DeepSpeed and the LBFGS optimizer are not compatible.

Error message

DeepSpeed and the LBFGS optimizer are not compatible.

What it means

The DeepSpeed precision plugin's optimizer_step found an LBFGS optimizer. DeepSpeed wraps optimizers in its own engine (DeepSpeedEngine) that performs fused steps with its internal gradient handling, and LBFGS's line-search closure re-evaluation breaks that contract, so Lightning rejects the combination.

Source

Thrown at src/lightning/pytorch/plugins/precision/deepspeed.py:128

        """
        if is_overridden("backward", model):
            warning_cache.warn(
                "You have overridden the `LightningModule.backward` hook but it will be ignored since DeepSpeed handles"
                " the backward logic internally."
            )
        deepspeed_engine: deepspeed.DeepSpeedEngine = model.trainer.model
        deepspeed_engine.backward(tensor, *args, **kwargs)

    @override
    def optimizer_step(  # type: ignore[override]
        self,
        optimizer: Steppable,
        model: "pl.LightningModule",
        closure: Callable[[], Any],
        **kwargs: Any,
    ) -> Any:
        if isinstance(optimizer, LBFGS):
            raise MisconfigurationException("DeepSpeed and the LBFGS optimizer are not compatible.")
        closure_result = closure()
        self._after_closure(model, optimizer)
        skipped_backward = closure_result is None
        # in manual optimization, the closure does not return a value
        if model.automatic_optimization and skipped_backward:
            raise MisconfigurationException(
                "Skipping backward by returning `None` from your `training_step` is not supported by `DeepSpeed`"
            )
        # DeepSpeed handles the optimizer step internally
        deepspeed_engine: deepspeed.DeepSpeedEngine = model.trainer.model
        return deepspeed_engine.step(**kwargs)

    @override
    def clip_gradients(
        self,
        optimizer: Optimizer,
        clip_val: Union[int, float] = 0.0,
        gradient_clip_algorithm: GradClipAlgorithmType = GradClipAlgorithmType.NORM,

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Replace LBFGS with a first-order optimizer (AdamW, SGD with momentum) for DeepSpeed runs
2. Run the LBFGS workload without DeepSpeed (strategy='auto'/'ddp', typically on CPU/single GPU)
3. If the line-search behavior is essential, implement a custom loop or useHigherOrderStrategy outside Lightning's DeepSpeed path

Example fix

# before
Trainer(strategy='deepspeed', precision='bf16-mixed')
# in LightningModule:
optimizer = torch.optim.LBFGS(self.parameters(), lr=0.5)

# after
Trainer(strategy='deepspeed', precision='bf16-mixed')
optimizer = torch.optim.AdamW(self.parameters(), lr=1e-3)

import torch

def deepspeed_compatible(optimizer) -> bool:
    return not isinstance(optimizer, torch.optim.LBFGS)

assert deepspeed_compatible(optimizer), 'LBFGS is unsupported with strategy=deepspeed'

def is_first_order_optimizer(opt) -> bool:
    import torch
    return not isinstance(opt, torch.optim.LBFGS)

When it happens

Trigger: Trainer(strategy='deepspeed', ...) with configure_optimizers returning torch.optim.LBFGS; optimizer_step receives the LBFGS instance before delegating to deepspeed_engine.step().

Common situations: Porting a single-GPU LBFGS training script to DeepSpeed multi-GPU; using second-order methods in distributed setups; tutorial configs mixing LBFGS with a distributed strategy.

Related errors

• AMP and the LBFGS optimizer are not compatible.
• AMP and the LBFGS optimizer are not compatible.
• `setup_optimizers` requires at least one optimizer as input.
• An optimizer should be passed only once to the `setup_optimi
• `{type(self).__name__}` does not own a group. HINT: try `col

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/a96620fec9796c24. Report an issue: GitHub.