{"record":{"id":"eb8393182660e154","repo":"Lightning-AI/pytorch-lightning","slug":"the-loss-returned-in-training-step-is-loss","errorCode":null,"errorMessage":"The loss returned in `training_step` is {loss}.","messagePattern":"The loss returned in `training_step` is (.+?)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"src/lightning/pytorch/loops/utilities.py","lineNumber":47,"sourceCode":"from lightning.pytorch.loops.fetchers import _DataFetcher, _DataLoaderIterDataFetcher, _PrefetchDataFetcher\nfrom lightning.pytorch.loops.progress import _BaseProgress\nfrom lightning.pytorch.strategies import FSDPStrategy\nfrom lightning.pytorch.strategies.parallel import ParallelStrategy\nfrom lightning.pytorch.strategies.strategy import Strategy\nfrom lightning.pytorch.trainer.states import RunningStage\nfrom lightning.pytorch.utilities.rank_zero import rank_zero_warn\nfrom lightning.pytorch.utilities.signature_utils import is_param_in_hook_signature\n\n\ndef check_finite_loss(loss: Optional[Tensor]) -> None:\n    \"\"\"Checks for finite loss value.\n\n    Args:\n        loss: the loss value to check to be finite\n\n    \"\"\"\n    if loss is not None and not torch.isfinite(loss).all():\n        raise ValueError(f\"The loss returned in `training_step` is {loss}.\")\n\n\ndef _parse_loop_limits(\n    min_steps: Optional[int],\n    max_steps: int,\n    min_epochs: Optional[int],\n    max_epochs: Optional[int],\n    trainer: \"pl.Trainer\",\n) -> tuple[int, int]:\n    \"\"\"This utility computes the default values for the minimum and maximum number of steps and epochs given the values\n    the user has selected.\n\n    Args:\n        min_steps: Minimum number of steps.\n        max_steps: Maximum number of steps.\n        min_epochs: Minimum number of epochs.\n        max_epochs: Maximum number of epochs.\n        trainer: Trainer instance.","sourceCodeStart":29,"sourceCodeEnd":65,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/pytorch/loops/utilities.py#L29-L65","documentation":"Raised by check_finite_loss when the loss tensor returned by training_step contains NaN or Inf values (torch.isfinite(loss).all() is False). This is a guard added on the training loop side so numerical blowups surface immediately instead of poisoning weights and producing garbage checkpoints.","triggerScenarios":"Loss overflow with fp16 (precision='16-mixed') and no GradScaler headroom; division by zero in a custom loss; exploding gradients from too-high learning rate; bad input data (NaNs in the batch) or NaN-producing ops like log(0).","commonSituations":"Switching to mixed precision and hitting fp16 range limits; unfixed NaNs in a data pipeline; unstable GAN training; log of zero from label-smoothed cross entropy without eps clamping.","solutions":["Clamp/epsilon-guard the loss computation (e.g. `torch.log(x + 1e-8)`)","Lower the learning rate or enable gradient clipping `Trainer(gradient_clip_val=1.0)`","Switch precision='16-mixed' to 'bf16-mixed' (wider dynamic range) if on supported hardware","Sanitize input batches: `torch.nan_to_num(batch)` or filter NaN samples in the dataset"],"exampleFix":"# before\nloss = -(y * torch.log(probs)).sum()  # log(0) -> -inf -> NaN loss\n\n# after\nloss = -(y * torch.log(probs.clamp_min(1e-8))).sum()","handlingStrategy":"validation","validationCode":"# in training_step, guard before returning\nif not torch.isfinite(loss):\n    loss = torch.nan_to_num(loss, nan=0.0, posinf=1e6, neginf=-1e6)  # or raise with batch context\nreturn loss","typeGuard":"def loss_is_finite(loss: torch.Tensor) -> bool:\n    return bool(torch.isfinite(loss).all())","tryCatchPattern":"try:\n    trainer.fit(model)\nexcept ValueError as e:\n    if 'training_step' in str(e) and 'is tensor' in str(e) or 'nan' in str(e).lower():\n        # drop/inspect the offending batch, lower LR, or switch to bf16\n        ...","preventionTips":["Epsilon-guard logs, softmax inputs, and divisions in custom losses","Enable Trainer(gradient_clip_val=...), track_for_nan, and monitor loss with detect_anomaly for debugging","Prefer bf16-mixed over 16-mixed when hardware supports it","Sanitize NaNs in the data pipeline before training"],"tags":["pytorch-lightning","nan-loss","numerical-stability","training"],"backgroundTag":"nan-loss-during-training","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}