{"record":{"id":"926d5409874087ca","repo":"Lightning-AI/pytorch-lightning","slug":"amp-and-the-lbfgs-optimizer-are-not-compatible","errorCode":null,"errorMessage":"AMP and the LBFGS optimizer are not compatible.","messagePattern":"AMP and the LBFGS optimizer are not compatible\\.","errorType":"validation","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"src/lightning/fabric/plugins/precision/amp.py","lineNumber":89,"sourceCode":"        return apply_to_collection(data, function=_convert_fp_tensor, dtype=Tensor, dst_type=torch.get_default_dtype())\n\n    @override\n    def backward(self, tensor: Tensor, model: Optional[Module], *args: Any, **kwargs: Any) -> None:\n        if self.scaler is not None:\n            tensor = self.scaler.scale(tensor)\n        super().backward(tensor, model, *args, **kwargs)\n\n    @override\n    def optimizer_step(\n        self,\n        optimizer: Optimizable,\n        **kwargs: Any,\n    ) -> Any:\n        if self.scaler is None:\n            # skip scaler logic, as bfloat16 does not require scaler\n            return super().optimizer_step(optimizer, **kwargs)\n        if isinstance(optimizer, LBFGS):\n            raise TypeError(\"AMP and the LBFGS optimizer are not compatible.\")\n        # note: the scaler will skip the `optimizer.step` if nonfinite gradients are found\n        step_output = self.scaler.step(optimizer, **kwargs)  # type: ignore[arg-type]\n        self.scaler.update()\n        return step_output\n\n    @override\n    def state_dict(self) -> dict[str, Any]:\n        if self.scaler is not None:\n            return self.scaler.state_dict()\n        return {}\n\n    @override\n    def load_state_dict(self, state_dict: dict[str, Any]) -> None:\n        if self.scaler is not None:\n            self.scaler.load_state_dict(state_dict)\n\n    @override\n    def unscale_gradients(self, optimizer: Optimizer) -> None:","sourceCodeStart":71,"sourceCodeEnd":107,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/plugins/precision/amp.py#L71-L107","documentation":"GradScaler-based AMP cannot wrap LBFGS because LBFGS's step performs multiple function evaluations, which the scaler's unscale/skip-if-inf/nan logic cannot handle. Lightning therefore raises TypeError when optimizer_step is called with an LBFGS optimizer while a scaler is active (i.e. '16-mixed').","triggerScenarios":"Using torch.optim.LBFGS with MixedPrecision(precision='16-mixed'); the error fires on the first optimizer step, not at setup.","commonSituations":"Porting LBFGS-based optimization (e.g. some classical/physics or GAN-style setups) into an AMP-enabled Fabric/Trainer run.","solutions":["Switch precision to 'bf16-mixed' (scaler is None, so the scaler path is skipped) or disable mixed precision","Replace LBFGS with a comparable multi-step optimizer (Adam, L-BFGS via a library that supports AMP)","Use a custom Precision plugin that implements optimizer_step for LBFGS without the scaler"],"exampleFix":"# before\nfabric = Fabric(precision=\"16-mixed\")  # + torch.optim.LBFGS\n\n# after\nfabric = Fabric(precision=\"bf16-mixed\")  # no scaler, LBFGS works","handlingStrategy":"validation","validationCode":"import torch\nfrom lightning.fabric.plugins.precision.amp import MixedPrecision\n\ndef check_amp_compatible(optimizer, plugin: MixedPrecision):\n    if plugin.scaler is not None and isinstance(optimizer, torch.optim.LBFGS):\n        raise SystemExit(\"LBFGS requires bf16-mixed or full precision\")","typeGuard":"def amp_lbfgs_ok(plugin, optimizer) -> bool:\n    return plugin.scaler is None or not isinstance(optimizer, torch.optim.LBFGS)","tryCatchPattern":"try:\n    step_out = plugin.optimizer_step(optimizer)\nexcept TypeError as e:\n    if \"LBFGS\" not in str(e):\n        raise\n    # switch precision and rebuild","preventionTips":["Avoid LBFGS under fp16 AMP; use bf16-mixed","Validate optimizer/plugin compatibility at setup time, not first step"],"tags":["amp","lbfgs","optimizer","pytorch-lightning"],"backgroundTag":"optimizer-amp-incompatible","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}