{"record":{"id":"a96620fec9796c24","repo":"Lightning-AI/pytorch-lightning","slug":"deepspeed-and-the-lbfgs-optimizer-are-not-compatib","errorCode":null,"errorMessage":"DeepSpeed and the LBFGS optimizer are not compatible.","messagePattern":"DeepSpeed and the LBFGS optimizer are not compatible\\.","errorType":"exception","errorClass":"MisconfigurationException","httpStatus":null,"severity":"error","filePath":"src/lightning/pytorch/plugins/precision/deepspeed.py","lineNumber":128,"sourceCode":"        \"\"\"\n        if is_overridden(\"backward\", model):\n            warning_cache.warn(\n                \"You have overridden the `LightningModule.backward` hook but it will be ignored since DeepSpeed handles\"\n                \" the backward logic internally.\"\n            )\n        deepspeed_engine: deepspeed.DeepSpeedEngine = model.trainer.model\n        deepspeed_engine.backward(tensor, *args, **kwargs)\n\n    @override\n    def optimizer_step(  # type: ignore[override]\n        self,\n        optimizer: Steppable,\n        model: \"pl.LightningModule\",\n        closure: Callable[[], Any],\n        **kwargs: Any,\n    ) -> Any:\n        if isinstance(optimizer, LBFGS):\n            raise MisconfigurationException(\"DeepSpeed and the LBFGS optimizer are not compatible.\")\n        closure_result = closure()\n        self._after_closure(model, optimizer)\n        skipped_backward = closure_result is None\n        # in manual optimization, the closure does not return a value\n        if model.automatic_optimization and skipped_backward:\n            raise MisconfigurationException(\n                \"Skipping backward by returning `None` from your `training_step` is not supported by `DeepSpeed`\"\n            )\n        # DeepSpeed handles the optimizer step internally\n        deepspeed_engine: deepspeed.DeepSpeedEngine = model.trainer.model\n        return deepspeed_engine.step(**kwargs)\n\n    @override\n    def clip_gradients(\n        self,\n        optimizer: Optimizer,\n        clip_val: Union[int, float] = 0.0,\n        gradient_clip_algorithm: GradClipAlgorithmType = GradClipAlgorithmType.NORM,","sourceCodeStart":110,"sourceCodeEnd":146,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/pytorch/plugins/precision/deepspeed.py#L110-L146","documentation":"The DeepSpeed precision plugin's optimizer_step found an LBFGS optimizer. DeepSpeed wraps optimizers in its own engine (DeepSpeedEngine) that performs fused steps with its internal gradient handling, and LBFGS's line-search closure re-evaluation breaks that contract, so Lightning rejects the combination.","triggerScenarios":"Trainer(strategy='deepspeed', ...) with configure_optimizers returning torch.optim.LBFGS; optimizer_step receives the LBFGS instance before delegating to deepspeed_engine.step().","commonSituations":"Porting a single-GPU LBFGS training script to DeepSpeed multi-GPU; using second-order methods in distributed setups; tutorial configs mixing LBFGS with a distributed strategy.","solutions":["Replace LBFGS with a first-order optimizer (AdamW, SGD with momentum) for DeepSpeed runs","Run the LBFGS workload without DeepSpeed (strategy='auto'/'ddp', typically on CPU/single GPU)","If the line-search behavior is essential, implement a custom loop or useHigherOrderStrategy outside Lightning's DeepSpeed path"],"exampleFix":"# before\nTrainer(strategy='deepspeed', precision='bf16-mixed')\n# in LightningModule:\noptimizer = torch.optim.LBFGS(self.parameters(), lr=0.5)\n\n# after\nTrainer(strategy='deepspeed', precision='bf16-mixed')\noptimizer = torch.optim.AdamW(self.parameters(), lr=1e-3)","handlingStrategy":"validation","validationCode":"import torch\n\ndef deepspeed_compatible(optimizer) -> bool:\n    return not isinstance(optimizer, torch.optim.LBFGS)\n\nassert deepspeed_compatible(optimizer), 'LBFGS is unsupported with strategy=deepspeed'","typeGuard":"def is_first_order_optimizer(opt) -> bool:\n    import torch\n    return not isinstance(opt, torch.optim.LBFGS)","tryCatchPattern":null,"preventionTips":["Smoke-test one trainer.fit step with the real optimizer before multi-node runs","Keep a strategy-optimizer compatibility checklist for LBFGS/second-order methods"],"tags":["pytorch-lightning","deepspeed","lbfgs","optimizer","distributed"],"backgroundTag":"optimizer-strategy-incompatibility","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}