Lightning-AI/pytorch-lightning · error · RuntimeError

The current optimizer, {type(optimizer).__qualname__}, does

Error message

The current optimizer, {type(optimizer).__qualname__}, does not allow for gradient clipping because it performs unscaling of gradients internally. HINT: Are you using a 'fused' optimizer?

What it means

Gradient clipping was requested (clip_val > 0) with an active GradScaler, but the optimizer performs its own unsscaling internally (detected via _optimizer_handles_unscaling, e.g. fused optimizers like fused AdamW). Since the AMP plugin clips gradients after unscaling them with its own scaler, double-unsscaling would corrupt the gradients, so the combination is rejected.

Source

Thrown at src/lightning/pytorch/plugins/precision/amp.py:127

        self._after_closure(model, optimizer)

        # in manual optimization, the closure does not return a value
        if not skip_unscaling:
            # note: the scaler will skip the `optimizer.step` if nonfinite gradients are found
            step_output = self.scaler.step(optimizer, **kwargs)  # type: ignore[arg-type]
            self.scaler.update()
            return step_output
        return closure_result

    @override
    def clip_gradients(
        self,
        optimizer: Optimizer,
        clip_val: Union[int, float] = 0.0,
        gradient_clip_algorithm: GradClipAlgorithmType = GradClipAlgorithmType.NORM,
    ) -> None:
        if clip_val > 0 and self.scaler is not None and _optimizer_handles_unscaling(optimizer):
            raise RuntimeError(
                f"The current optimizer, {type(optimizer).__qualname__}, does not allow for gradient clipping"
                " because it performs unscaling of gradients internally. HINT: Are you using a 'fused' optimizer?"
            )
        super().clip_gradients(optimizer=optimizer, clip_val=clip_val, gradient_clip_algorithm=gradient_clip_algorithm)

    def autocast_context_manager(self) -> torch.autocast:
        dtype = torch.bfloat16 if self.precision == "bf16-mixed" else torch.half
        return torch.autocast(self.device, dtype=dtype, cache_enabled=False)

    @override
    @contextmanager
    def forward_context(self) -> Generator[None, None, None]:
        """Enable autocast and clear cached casts after nested grad-disabling contexts exit."""
        original_no_grad = torch.no_grad
        original_inference_mode = torch.inference_mode

        def _clear_cache_on_exit(
            context_factory: Callable[..., Any], *, clear_cache: Callable[..., bool]

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Drop fused=True from the optimizer so the GradScaler handles unscaling and clipping
2. Keep the fused optimizer but switch precision to 'bf16-mixed' (no scaler, clipping allowed)
3. Disable gradient clipping (set Trainer(gradient_clip_val=0) or leave it None)

Example fix

# before
optimizer = torch.optim.AdamW(self.parameters(), lr=1e-3, fused=True)
Trainer(precision='16-mixed', gradient_clip_val=1.0)

# after
optimizer = torch.optim.AdamW(self.parameters(), lr=1e-3)
Trainer(precision='16-mixed', gradient_clip_val=1.0)

def optimizer_allows_amp_clipping(optimizer) -> bool:
    # fused optimizers unscale internally and cannot be clipped by the GradScaler
    return not getattr(optimizer, 'fused', False)

if trainer.gradient_clip_val and trainer.precision == '16-mixed':
    assert optimizer_allows_amp_clipping(optimizer)

def is_fused_optimizer(opt) -> bool:
    return bool(getattr(opt, 'fused', False)) or type(opt).__name__ in {'FusedAdam','FusedSGD'}

try:
    trainer.fit(model)
except RuntimeError as e:
    if 'does not allow for gradient clipping' in str(e):
        optimizer = rebuild_without_fused(optimizer); trainer.fit(model)
    else:
        raise

When it happens

Trigger: Trainer(precision='16-mixed', gradient_clip_val=1.0) with optimizer = torch.optim.AdamW(..., fused=True) (or any optimizer whose step handles unscaling); the plugin's clip_gradients is then called with a scaler present.

Common situations: Enabling fused=True for performance on newer GPUs while keeping AMP fp16 and gradient clipping; upgrading PyTorch where fused AdamW became common; following performance-tuning guides that recommend fused optimizers.

Related errors

• Gradient clipping is not implemented for optimizers handling
• `precision='bf16-mixed'` does not use a scaler, found {scale
• `precision='bf16-mixed'` does not use a scaler, found {scale
• AMP and the LBFGS optimizer are not compatible.
• Passed `{type(self).__name__}(precision={precision!r})`. Pre

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/c3ee1b4010b3c3a0. Report an issue: GitHub.