{"record":{"id":"d99e00bb4d758407","repo":"Lightning-AI/pytorch-lightning","slug":"gradient-clipping-is-not-implemented-for-optimizer","errorCode":null,"errorMessage":"Gradient clipping is not implemented for optimizers handling the unscaling.","messagePattern":"Gradient clipping is not implemented for optimizers handling the unscaling\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/lightning/fabric/plugins/precision/amp.py","lineNumber":111,"sourceCode":"        return step_output\n\n    @override\n    def state_dict(self) -> dict[str, Any]:\n        if self.scaler is not None:\n            return self.scaler.state_dict()\n        return {}\n\n    @override\n    def load_state_dict(self, state_dict: dict[str, Any]) -> None:\n        if self.scaler is not None:\n            self.scaler.load_state_dict(state_dict)\n\n    @override\n    def unscale_gradients(self, optimizer: Optimizer) -> None:\n        scaler = self.scaler\n        if scaler is not None:\n            if _optimizer_handles_unscaling(optimizer):\n                raise NotImplementedError(\"Gradient clipping is not implemented for optimizers handling the unscaling.\")\n            scaler.unscale_(optimizer)\n\n\ndef _optimizer_handles_unscaling(optimizer: Any) -> bool:\n    \"\"\"Determines whether a PyTorch optimizer handles unscaling gradients in the step method rather than through the\n    :class:`torch.cuda.amp.GradScaler`.\n\n    Since, the current implementation of this function checks a PyTorch internal variable on the optimizer, the return\n    value will only be reliable for built-in PyTorch optimizers.\n\n    \"\"\"\n    return getattr(optimizer, \"_step_supports_amp_scaling\", False)\n","sourceCodeStart":93,"sourceCodeEnd":124,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/plugins/precision/amp.py#L93-L124","documentation":"unscale_gradients calls scaler.unscale_(optimizer), but some optimizers (e.g. OptimizerWithScalerWrapper variants / fused optimizers that handle unscaling internally in step) cannot be unscaled externally. Gradient clipping is not implemented for such optimizers under scaler-based AMP.","triggerScenarios":"MixedPrecision(precision='16-mixed') with gradient clipping enabled (clip_gradients) while the optimizer's step handles unscaling itself (detected by _optimizer_handles_unscaling, e.g. optimizers with a built-in scaler).","commonSituations":"Using fused/custom optimizers that integrate the GradScaler (some fairseq/apex-style wrappers) together with Lightning's gradient clipping on fp16-mixed.","solutions":["Switch to 'bf16-mixed' so no scaler/unscale path is used","Use an optimizer that lets GradScaler.unscale_ handle unscaling (plain torch optimizers)","Disable gradient clipping or implement clipping inside the optimizer's step"],"exampleFix":"# before\nfabric = Fabric(precision=\"16-mixed\")\nfabric.clip_gradients(model, optimizer, clip_val=1.0)  # optimizer self-uncales\n\n# after\nfabric = Fabric(precision=\"bf16-mixed\")\nfabric.clip_gradients(model, optimizer, clip_val=1.0)","handlingStrategy":"validation","validationCode":"from lightning.fabric.plugins.precision.amp import _optimizer_handles_unscaling\n\ndef clipping_supported(plugin, optimizer) -> bool:\n    return plugin.scaler is None or not _optimizer_handles_unscaling(optimizer)","typeGuard":"def can_clip_gradients(plugin, optimizer) -> bool:\n    from lightning.fabric.plugins.precision.amp import _optimizer_handles_unscaling\n    return plugin.scaler is None or not _optimizer_handles_unscaling(optimizer)","tryCatchPattern":null,"preventionTips":["Prefer bf16-mixed when using fused/self-unscaling optimizers","Check optimizer type before enabling gradient clipping"],"tags":["amp","gradient-clipping","fused-optimizer","pytorch-lightning"],"backgroundTag":"unsupported-operation-amp-clipping","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}