Lightning-AI/pytorch-lightning · error · MisconfigurationException

AMP and the LBFGS optimizer are not compatible.

Error message

AMP and the LBFGS optimizer are not compatible.

What it means

The AMP precision plugin's optimizer_step detects an LBFGS optimizer while a GradScaler is active (16-mixed). LBFGS re-evaluates the closure multiple times per step, which is incompatible with the single unscale/step cadence that torch.amp.GradScaler enforces. Lightning therefore refuses the combination outright.

Source

Thrown at src/lightning/pytorch/plugins/precision/amp.py:97

    @override
    def pre_backward(self, tensor: Tensor, module: "pl.LightningModule") -> Tensor:  # type: ignore[override]
        if self.scaler is not None:
            tensor = self.scaler.scale(tensor)
        return super().pre_backward(tensor, module)

    @override
    def optimizer_step(  # type: ignore[override]
        self,
        optimizer: Optimizable,
        model: "pl.LightningModule",
        closure: Callable[[], Any],
        **kwargs: Any,
    ) -> Any:
        if self.scaler is None:
            # skip scaler logic, as bfloat16 does not require scaler
            return super().optimizer_step(optimizer, model=model, closure=closure, **kwargs)
        if isinstance(optimizer, LBFGS):
            raise MisconfigurationException("AMP and the LBFGS optimizer are not compatible.")
        closure_result = closure()

        # If backward was skipped in automatic optimization (return None), unscaling is not needed
        skip_unscaling = closure_result is None and model.automatic_optimization

        if not _optimizer_handles_unscaling(optimizer) and not skip_unscaling:
            # Unscaling needs to be performed here in case we are going to apply gradient clipping.
            # Optimizers that perform unscaling in their `.step()` method are not supported (e.g., fused Adam).
            # Note: `unscale` happens after the closure is executed, but before the `on_before_optimizer_step` hook.
            self.scaler.unscale_(optimizer)  # type: ignore[arg-type]

        self._after_closure(model, optimizer)

        # in manual optimization, the closure does not return a value
        if not skip_unscaling:
            # note: the scaler will skip the `optimizer.step` if nonfinite gradients are found
            step_output = self.scaler.step(optimizer, **kwargs)  # type: ignore[arg-type]
            self.scaler.update()

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Switch precision to '32-true' (or 'bf16-mixed', which skips the scaler) when using LBFGS
2. Replace LBFGS with Adam/AdamW/SGD if you must keep 16-mixed AMP
3. Pass scaler=None to the plugin and rely on bf16 or fp32

Example fix

# before
Trainer(precision='16-mixed', max_epochs=100)
optimizer = torch.optim.LBFGS(self.parameters(), lr=1)

# after
Trainer(precision='32-true', max_epochs=100)
optimizer = torch.optim.LBFGS(self.parameters(), lr=1)

import torch

def amp_compatible(optimizer) -> bool:
    return not isinstance(optimizer, torch.optim.LBFGS)

# before Trainer fit with precision='16-mixed':
assert amp_compatible(optimizer), 'LBFGS requires precision 32-true or bf16-mixed'

from torch.optim import Optimizer, LBFGS

def uses_scaler_safe_step(opt: Optimizer) -> bool:
    return not isinstance(opt, LBFGS)

When it happens

Trigger: Trainer(precision='16-mixed', plugins=[MixedPrecisionPlugin(...)]) together with torch.optim.LBFGS as the model's optimizer; optimizer_step is then called with an LBFGS instance while self.scaler is not None.

Common situations: Using LBFGS (e.g. for small full-batch fits or physics-informed ML) with default 16-mixed precision on GPU; converting a CPU 32-true script to GPU AMP; copying an LBFGS example into an AMP training template.

Related errors

• AMP and the LBFGS optimizer are not compatible.
• `precision='bf16-mixed'` does not use a scaler, found {scale
• `precision='bf16-mixed'` does not use a scaler, found {scale
• The current optimizer, {type(optimizer).__qualname__}, does
• DeepSpeed and the LBFGS optimizer are not compatible.

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/cc3adca0ee48da56. Report an issue: GitHub.