{"record":{"id":"cc3adca0ee48da56","repo":"Lightning-AI/pytorch-lightning","slug":"amp-and-the-lbfgs-optimizer-are-not-compatible-cc3adc","errorCode":null,"errorMessage":"AMP and the LBFGS optimizer are not compatible.","messagePattern":"AMP and the LBFGS optimizer are not compatible\\.","errorType":"exception","errorClass":"MisconfigurationException","httpStatus":null,"severity":"error","filePath":"src/lightning/pytorch/plugins/precision/amp.py","lineNumber":97,"sourceCode":"    @override\n    def pre_backward(self, tensor: Tensor, module: \"pl.LightningModule\") -> Tensor:  # type: ignore[override]\n        if self.scaler is not None:\n            tensor = self.scaler.scale(tensor)\n        return super().pre_backward(tensor, module)\n\n    @override\n    def optimizer_step(  # type: ignore[override]\n        self,\n        optimizer: Optimizable,\n        model: \"pl.LightningModule\",\n        closure: Callable[[], Any],\n        **kwargs: Any,\n    ) -> Any:\n        if self.scaler is None:\n            # skip scaler logic, as bfloat16 does not require scaler\n            return super().optimizer_step(optimizer, model=model, closure=closure, **kwargs)\n        if isinstance(optimizer, LBFGS):\n            raise MisconfigurationException(\"AMP and the LBFGS optimizer are not compatible.\")\n        closure_result = closure()\n\n        # If backward was skipped in automatic optimization (return None), unscaling is not needed\n        skip_unscaling = closure_result is None and model.automatic_optimization\n\n        if not _optimizer_handles_unscaling(optimizer) and not skip_unscaling:\n            # Unscaling needs to be performed here in case we are going to apply gradient clipping.\n            # Optimizers that perform unscaling in their `.step()` method are not supported (e.g., fused Adam).\n            # Note: `unscale` happens after the closure is executed, but before the `on_before_optimizer_step` hook.\n            self.scaler.unscale_(optimizer)  # type: ignore[arg-type]\n\n        self._after_closure(model, optimizer)\n\n        # in manual optimization, the closure does not return a value\n        if not skip_unscaling:\n            # note: the scaler will skip the `optimizer.step` if nonfinite gradients are found\n            step_output = self.scaler.step(optimizer, **kwargs)  # type: ignore[arg-type]\n            self.scaler.update()","sourceCodeStart":79,"sourceCodeEnd":115,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/pytorch/plugins/precision/amp.py#L79-L115","documentation":"The AMP precision plugin's optimizer_step detects an LBFGS optimizer while a GradScaler is active (16-mixed). LBFGS re-evaluates the closure multiple times per step, which is incompatible with the single unscale/step cadence that torch.amp.GradScaler enforces. Lightning therefore refuses the combination outright.","triggerScenarios":"Trainer(precision='16-mixed', plugins=[MixedPrecisionPlugin(...)]) together with torch.optim.LBFGS as the model's optimizer; optimizer_step is then called with an LBFGS instance while self.scaler is not None.","commonSituations":"Using LBFGS (e.g. for small full-batch fits or physics-informed ML) with default 16-mixed precision on GPU; converting a CPU 32-true script to GPU AMP; copying an LBFGS example into an AMP training template.","solutions":["Switch precision to '32-true' (or 'bf16-mixed', which skips the scaler) when using LBFGS","Replace LBFGS with Adam/AdamW/SGD if you must keep 16-mixed AMP","Pass scaler=None to the plugin and rely on bf16 or fp32"],"exampleFix":"# before\nTrainer(precision='16-mixed', max_epochs=100)\noptimizer = torch.optim.LBFGS(self.parameters(), lr=1)\n\n# after\nTrainer(precision='32-true', max_epochs=100)\noptimizer = torch.optim.LBFGS(self.parameters(), lr=1)","handlingStrategy":"validation","validationCode":"import torch\n\ndef amp_compatible(optimizer) -> bool:\n    return not isinstance(optimizer, torch.optim.LBFGS)\n\n# before Trainer fit with precision='16-mixed':\nassert amp_compatible(optimizer), 'LBFGS requires precision 32-true or bf16-mixed'","typeGuard":"from torch.optim import Optimizer, LBFGS\n\ndef uses_scaler_safe_step(opt: Optimizer) -> bool:\n    return not isinstance(opt, LBFGS)","tryCatchPattern":null,"preventionTips":["Keep a compatibility matrix: LBFGS <-> fp32/bf16 only","Add a unit test asserting configure_optimizers output is AMP-compatible when precision is 16-mixed"],"tags":["pytorch-lightning","amp","lbfgs","optimizer","gradscaler","optimizer-step"],"backgroundTag":"optimizer-amp-incompatibility","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}