Lightning-AI/pytorch-lightning · error · MisconfigurationException

`gradient_clip_algorithm='norm'` is currently not supported

Error message

`gradient_clip_algorithm='norm'` is currently not supported for `{self.__class__.__name__}`

What it means

FSDPMixedPrecisionPlugin.clip_grad_by_norm unconditionally raises because torch.nn.utils.clip_grad_norm_ is mathematically wrong for sharded FSDP parameters (norms must be reduced across shards). The correct API is the root module's clip_grad_norm, but the plugin has no reference to the root module, so Lightning blocks norm-based clipping for this plugin class instead of silently clipping incorrectly.

Source

Thrown at src/lightning/pytorch/plugins/precision/fsdp.py:90

            "bf16-true": torch.bfloat16,
            "16-true": torch.float16,
            "32-true": torch.float32,
        }
        self._desired_input_dtype = precision_to_type[self.precision]

    @override
    def convert_module(self, module: Module) -> Module:
        if "true" in self.precision:
            return module.to(dtype=self._desired_input_dtype)
        return module

    @override
    def clip_grad_by_norm(self, *_: Any, **__: Any) -> None:
        # see https://pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.FullyShardedDataParallel.clip_grad_norm_
        # section `Gradient Clipping`, using `torch.nn.utils.clip_grad_norm_` is incorrect with FSDP.
        # To overcome this we need to call root_sharded_module.clip_grad_norm(clip_val), but we don't have a reference
        # to the root module
        raise MisconfigurationException(
            f"`gradient_clip_algorithm='norm'` is currently not supported for `{self.__class__.__name__}`"
        )

    @property
    def mixed_precision_config(self) -> "TorchMixedPrecision":
        from torch.distributed.fsdp.fully_sharded_data_parallel import MixedPrecision as TorchMixedPrecision

        if self.precision in ("16-true", "bf16-true"):
            rank_zero_warn(
                f"FSDP with `{self.precision}` enables computation in lower precision. "
                "FSDP will always retain a full-precision copy of the model parameters for sharding."
            )

        if self.precision in ("16-true", "16-mixed"):
            param_dtype = reduce_dtype = buffer_dtype = torch.float16
        elif self.precision in ("bf16-true", "bf16-mixed"):
            param_dtype = reduce_dtype = buffer_dtype = torch.bfloat16
        elif self.precision == "32-true":

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Switch to value-based clipping: Trainer(..., gradient_clip_algorithm='value')
2. Use FSDP's own clipping via strategy configuration where supported (FSDP2 / newer Lightning exposes it)
3. Disable gradient clipping for the FSDP run

Example fix

# before
Trainer(strategy='fsdp', gradient_clip_val=0.5)  # defaults to algorithm='norm'

# after
from lightning.pytorch.callbacks import GradientClipAlgorithmType  # or string
Trainer(strategy='fsdp', gradient_clip_val=0.5, gradient_clip_algorithm='value')

from lightning.pytorch.trainer import Trainer

def clipped_fsdp_config(clip_val):
    return dict(strategy='fsdp', gradient_clip_val=clip_val,
                gradient_clip_algorithm='value')

trainer = Trainer(**clipped_fsdp_config(0.5))

def clip_algorithm_supported_for_fsdp(alg: str) -> bool:
    return alg == 'value'

When it happens

Trigger: Trainer(strategy='fsdp', gradient_clip_val=0.5, gradient_clip_algorithm='norm') (the default algorithm) with the FSDP mixed-precision plugin; the base clip_gradients dispatches to clip_grad_by_norm which raises.

Common situations: Copying gradient_clip_val from a DDP config into an FSDP run; default Trainer gradient_clip_algorithm being NORM so simply setting gradient_clip_val triggers it; FSDP1-based strategies in Lightning (FSDPStrategy/fully sharded plugins).

Related errors

• Found multiple FSDP models in the given state. Saving checkp
• Found multiple FSDP models in the given state. Loading check
• The hybrid sharding strategy requires you to pass at least o
• `precision={precision!r})` is not supported in FSDP. `precis
• The optimizer has references to the model's meta-device para

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/eda450983f6a4b1e. Report an issue: GitHub.