{"record":{"id":"8d8e5ca5fbf202a6","repo":"deepfakes/faceswap","slug":"a-nan-was-detected-and-you-have-nan-protection-ena","errorCode":null,"errorMessage":"A NaN was detected and you have NaN protection enabled. Training has been terminated.","messagePattern":"A NaN was detected and you have NaN protection enabled\\. Training has been terminated\\.","errorType":"exception","errorClass":"FaceswapError","httpStatus":null,"severity":"critical","filePath":"lib/training/train.py","lineNumber":376,"sourceCode":"        Raises\n        ------\n        FaceswapError\n            If a NaN is detected, a :class:`FaceswapError` will be raised\n        \"\"\"\n        # NaN protection\n        if mod_cfg.nan_protection() and not all(torch.isfinite(val.total).all() for val in loss):\n            loss_str = \", \".join(f\"Loss {get_label(i, len(loss))}: {round(x.total.item(), 6)}\"\n                                 for i, x in enumerate(loss))\n            msg = f\"NaN Detected. {loss_str}\"\n            failed = \", \".join(f\"{key}({get_label(i, len(loss))})\"\n                               for i, out in enumerate(loss)\n                               for unweighted in out.unweighted\n                               for key, sub_loss in unweighted.items()\n                               if not torch.isfinite(sub_loss).all())\n            if failed:\n                msg += f\". The loss function(s) that NaN'd: {failed}\"\n            logger.critical(msg)\n            raise FaceswapError(\"A NaN was detected and you have NaN protection enabled. Training \"\n                                \"has been terminated.\")\n\n        combined_loss = np.array([x.total.item() for x in loss], dtype=np.float32)\n        self._model.add_history(combined_loss)\n        logger.trace(\"[Trainer] original loss: %s, combined_loss: %s\",  # type:ignore[attr-defined]\n                     loss, combined_loss)\n        return combined_loss\n\n    def _print_loss(self, loss: np.ndarray) -> None:\n        \"\"\"Outputs the loss for the current iteration to the console.\n\n        Parameters\n        ----------\n        The loss for each side. List should contain 2 ``floats`` side \"a\" in position 0 and side\n        \"b\" in position 1.\n         \"\"\"\n        output = \", \".join([f\"Loss {side}: {side_loss:.5f}\"\n                            for side, side_loss in zip((\"A\", \"B\"), loss)])","sourceCodeStart":358,"sourceCodeEnd":394,"githubUrl":"https://github.com/deepfakes/faceswap/blob/f530cb7508ae670f6474f8a7d9c4df94705cf96b/lib/training/train.py#L358-L394","documentation":"Faceswap's NaN protection monitors every loss tensor each iteration; if any total or sub-loss becomes non-finite (NaN/Inf) while nan_protection is enabled in the training settings, training is deliberately terminated with this FaceswapError. The preceding log line names which loss component (labelled A/B per side) went non-finite.","triggerScenarios":"A training step produces NaN in any loss (loss.total or any unweighted sub-loss) while cfg nan_protection() is true. Common numerical causes: learning rate too high causing divergence, AMP fp16 overflow, or bad input data (corrupt/blank feeds).","commonSituations":"Raising the learning rate too aggressively; enabling mixed precision on a model sensitive to fp16; training on datasets containing misaligned/empty faces; resuming from a bad state.","solutions":["Lower the learning rate in the optimizer settings (e.g. halve it) and restart training from the last good snapshot.","If using mixed precision, disable it (or vice versa, try enabling it) to rule out fp16 overflow.","Check the log line naming the failing loss function and inspect the corresponding side's training faces for corrupt or empty images; re-extract if needed.","As a last resort, disable NaN protection in the training settings so training continues (only to salvage a session — the underlying divergence still needs fixing).","Restore from a recent model backup if the model state itself is corrupted."],"exampleFix":"# before\n[optimizer.learning_rate] = 5e-4\n\n# after\n[optimizer.learning_rate] = 1e-4  # halve/step down until stable","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"from lib.exceptions import FaceswapError\ntry:\n    trainer.train_one_iteration(...)\nexcept FaceswapError as err:\n    if \"NaN protection\" in str(err):\n        # halve LR, restore last snapshot, restart\n        adjust_learning_rate(0.5)\n        io.restore_snapshot()\n    else:\n        raise","preventionTips":["Use conservative learning rates when resuming or transferring models.","Watch early iterations for loss spikes (loss history/TensorBoard) so you can intervene before NaN triggers termination.","Validate the training set for corrupt/empty aligned faces before long runs."],"tags":["faceswap","training","nan","numerical-stability","cuda"],"backgroundTag":null,"analyzedSha":"f530cb7508ae670f6474f8a7d9c4df94705cf96b","analyzedAt":"2026-08-15T02:59:26.626Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}