{"record":{"id":"bf59ff592c0a40b8","repo":"Lightning-AI/pytorch-lightning","slug":"gradient-clipping-is-not-implemented-for-optimizer-bf59ff","errorCode":null,"errorMessage":"Gradient clipping is not implemented for optimizers handling the unscaling.","messagePattern":"Gradient clipping is not implemented for optimizers handling the unscaling\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/lightning/fabric/plugins/precision/fsdp.py","lineNumber":156,"sourceCode":"    def optimizer_step(\n        self,\n        optimizer: Optimizable,\n        **kwargs: Any,\n    ) -> Any:\n        if self.scaler is None:\n            # skip scaler logic, as bfloat16 does not require scaler\n            return super().optimizer_step(optimizer, **kwargs)\n        # note: the scaler will skip the `optimizer.step` if nonfinite gradients are found\n        step_output = self.scaler.step(optimizer, **kwargs)  # type: ignore[arg-type]\n        self.scaler.update()\n        return step_output\n\n    @override\n    def unscale_gradients(self, optimizer: Optimizer) -> None:\n        scaler = self.scaler\n        if scaler is not None:\n            if _optimizer_handles_unscaling(optimizer):\n                raise NotImplementedError(\"Gradient clipping is not implemented for optimizers handling the unscaling.\")\n            scaler.unscale_(optimizer)\n\n    @override\n    def state_dict(self) -> dict[str, Any]:\n        if self.scaler is not None:\n            return self.scaler.state_dict()\n        return {}\n\n    @override\n    def load_state_dict(self, state_dict: dict[str, Any]) -> None:\n        if self.scaler is not None:\n            self.scaler.load_state_dict(state_dict)\n","sourceCodeStart":138,"sourceCodeEnd":169,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/plugins/precision/fsdp.py#L138-L169","documentation":"FSDPPrecision.unscale_gradients calls scaler.unscale_(optimizer) when a GradScaler is active, but some optimizers (e.g. those with built-in gradient scoping like bnb optimizers) handle unscaling themselves (detected via _optimizer_handles_unscaling). Combining an external scaler with such an optimizer is unsupported, so a NotImplementedError is raised.","triggerScenarios":"Using FSDPPrecision with '16-mixed' (so a ShardedGradScaler exists) together with an optimizer whose class implements its own unscaling — e.g. bitsandbytes optimizers — which triggers _optimizer_handles_unscaling(optimizer) to return True when unscale_gradients is called.","commonSituations":"Mixing bitsandbytes 8-bit/16-bit optimizers with FSDP 16-mixed training; QLoRA-style setups moved under Fabric FSDP.","solutions":["Switch precision to 'bf16-mixed' (or 32-true) so no scaler is created and unscale_gradients becomes a no-op","Use an optimizer that does not implement its own unscaling (plain torch.optim) with '16-mixed'","If you hit this during clipping only, disable the fabric-side clipping for that optimizer"],"exampleFix":"# before\nFabric(precision=\"16-mixed\", strategy=FSDPPStrategy(), plugins=[...])  # with bnb optimizer\n# after\nFabric(precision=\"bf16-mixed\", strategy=FSDPPStrategy())  # bnb optimizer, no scaler","handlingStrategy":"fallback","validationCode":"from lightning.fabric.plugins.precision.fsdp import _optimizer_handles_unscaling\nuses_scaler = precision_plugin.scaler is not None\nif uses_scaler and _optimizer_handles_unscaling(optimizer):\n    raise RuntimeError(\"switch to bf16-mixed or a plain optimizer\")","typeGuard":"from lightning.fabric.plugins.precision.fsdp import _optimizer_handles_unscaling\n\ndef needs_external_unscaling(optimizer) -> bool:\n    return not _optimizer_handles_unscaling(optimizer)","tryCatchPattern":"try:\n    fabric.strategy.precision_plugin.unscale_gradients(optimizer)\nexcept NotImplementedError:\n    pass  # optimizer handles unscaling internally","preventionTips":["Pair bnb-style optimizers with bf16-mixed under FSDP","Keep a compatibility matrix of optimizer vs precision in your training utils"],"tags":["fsdp","grad-scaler","optimizer-incompatibility","pytorch-lightning"],"backgroundTag":"unsupported-operation-combination","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}