Lightning-AI/pytorch-lightning · critical · ValueError

The loss returned in `training_step` is {loss}.

Error message

The loss returned in `training_step` is {loss}.

What it means

Raised by check_finite_loss when the loss tensor returned by training_step contains NaN or Inf values (torch.isfinite(loss).all() is False). This is a guard added on the training loop side so numerical blowups surface immediately instead of poisoning weights and producing garbage checkpoints.

Source

Thrown at src/lightning/pytorch/loops/utilities.py:47

from lightning.pytorch.loops.fetchers import _DataFetcher, _DataLoaderIterDataFetcher, _PrefetchDataFetcher
from lightning.pytorch.loops.progress import _BaseProgress
from lightning.pytorch.strategies import FSDPStrategy
from lightning.pytorch.strategies.parallel import ParallelStrategy
from lightning.pytorch.strategies.strategy import Strategy
from lightning.pytorch.trainer.states import RunningStage
from lightning.pytorch.utilities.rank_zero import rank_zero_warn
from lightning.pytorch.utilities.signature_utils import is_param_in_hook_signature


def check_finite_loss(loss: Optional[Tensor]) -> None:
    """Checks for finite loss value.

    Args:
        loss: the loss value to check to be finite

    """
    if loss is not None and not torch.isfinite(loss).all():
        raise ValueError(f"The loss returned in `training_step` is {loss}.")


def _parse_loop_limits(
    min_steps: Optional[int],
    max_steps: int,
    min_epochs: Optional[int],
    max_epochs: Optional[int],
    trainer: "pl.Trainer",
) -> tuple[int, int]:
    """This utility computes the default values for the minimum and maximum number of steps and epochs given the values
    the user has selected.

    Args:
        min_steps: Minimum number of steps.
        max_steps: Maximum number of steps.
        min_epochs: Minimum number of epochs.
        max_epochs: Maximum number of epochs.
        trainer: Trainer instance.

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Clamp/epsilon-guard the loss computation (e.g. `torch.log(x + 1e-8)`)
2. Lower the learning rate or enable gradient clipping `Trainer(gradient_clip_val=1.0)`
3. Switch precision='16-mixed' to 'bf16-mixed' (wider dynamic range) if on supported hardware
4. Sanitize input batches: `torch.nan_to_num(batch)` or filter NaN samples in the dataset

Example fix

# before
loss = -(y * torch.log(probs)).sum()  # log(0) -> -inf -> NaN loss

# after
loss = -(y * torch.log(probs.clamp_min(1e-8))).sum()

# in training_step, guard before returning
if not torch.isfinite(loss):
    loss = torch.nan_to_num(loss, nan=0.0, posinf=1e6, neginf=-1e6)  # or raise with batch context
return loss

def loss_is_finite(loss: torch.Tensor) -> bool:
    return bool(torch.isfinite(loss).all())

try:
    trainer.fit(model)
except ValueError as e:
    if 'training_step' in str(e) and 'is tensor' in str(e) or 'nan' in str(e).lower():
        # drop/inspect the offending batch, lower LR, or switch to bf16
        ...

When it happens

Trigger: Loss overflow with fp16 (precision='16-mixed') and no GradScaler headroom; division by zero in a custom loss; exploding gradients from too-high learning rate; bad input data (NaNs in the batch) or NaN-producing ops like log(0).

Common situations: Switching to mixed precision and hitting fp16 range limits; unfixed NaNs in a data pipeline; unstable GAN training; log of zero from label-smoothed cross entropy without eps clamping.

Related errors

• `{type(self).__name__}` does not support the `CombinedLoader
• Device should be CPU, got {device} instead.
• `devices` selected with `CPUAccelerator` should be an int >
• Device should be CUDA, got {device} instead.
• You requested to find {num_devices} devices but there are no

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/eb8393182660e154. Report an issue: GitHub.