deepfakes/faceswap · critical · FaceswapError

A NaN was detected and you have NaN protection enabled. Trai

Error message

A NaN was detected and you have NaN protection enabled. Training has been terminated.

What it means

Faceswap's NaN protection monitors every loss tensor each iteration; if any total or sub-loss becomes non-finite (NaN/Inf) while nan_protection is enabled in the training settings, training is deliberately terminated with this FaceswapError. The preceding log line names which loss component (labelled A/B per side) went non-finite.

Source

Thrown at lib/training/train.py:376

        Raises
        ------
        FaceswapError
            If a NaN is detected, a :class:`FaceswapError` will be raised
        """
        # NaN protection
        if mod_cfg.nan_protection() and not all(torch.isfinite(val.total).all() for val in loss):
            loss_str = ", ".join(f"Loss {get_label(i, len(loss))}: {round(x.total.item(), 6)}"
                                 for i, x in enumerate(loss))
            msg = f"NaN Detected. {loss_str}"
            failed = ", ".join(f"{key}({get_label(i, len(loss))})"
                               for i, out in enumerate(loss)
                               for unweighted in out.unweighted
                               for key, sub_loss in unweighted.items()
                               if not torch.isfinite(sub_loss).all())
            if failed:
                msg += f". The loss function(s) that NaN'd: {failed}"
            logger.critical(msg)
            raise FaceswapError("A NaN was detected and you have NaN protection enabled. Training "
                                "has been terminated.")

        combined_loss = np.array([x.total.item() for x in loss], dtype=np.float32)
        self._model.add_history(combined_loss)
        logger.trace("[Trainer] original loss: %s, combined_loss: %s",  # type:ignore[attr-defined]
                     loss, combined_loss)
        return combined_loss

    def _print_loss(self, loss: np.ndarray) -> None:
        """Outputs the loss for the current iteration to the console.

        Parameters
        ----------
        The loss for each side. List should contain 2 ``floats`` side "a" in position 0 and side
        "b" in position 1.
         """
        output = ", ".join([f"Loss {side}: {side_loss:.5f}"
                            for side, side_loss in zip(("A", "B"), loss)])

View on GitHub (pinned to f530cb7508)

Solutions

Lower the learning rate in the optimizer settings (e.g. halve it) and restart training from the last good snapshot.
If using mixed precision, disable it (or vice versa, try enabling it) to rule out fp16 overflow.
Check the log line naming the failing loss function and inspect the corresponding side's training faces for corrupt or empty images; re-extract if needed.
As a last resort, disable NaN protection in the training settings so training continues (only to salvage a session — the underlying divergence still needs fixing).
Restore from a recent model backup if the model state itself is corrupted.

Example fix

# before
[optimizer.learning_rate] = 5e-4

# after
[optimizer.learning_rate] = 1e-4  # halve/step down until stable

Defensive patterns

Strategy: try-catch

Try / catch

from lib.exceptions import FaceswapError
try:
    trainer.train_one_iteration(...)
except FaceswapError as err:
    if "NaN protection" in str(err):
        # halve LR, restore last snapshot, restart
        adjust_learning_rate(0.5)
        io.restore_snapshot()
    else:
        raise

Prevention

Use conservative learning rates when resuming or transferring models.
Watch early iterations for loss spikes (loss history/TensorBoard) so you can intervene before NaN triggers termination.
Validate the training set for corrupt/empty aligned faces before long runs.

When it happens

Trigger: A training step produces NaN in any loss (loss.total or any unweighted sub-loss) while cfg nan_protection() is true. Common numerical causes: learning rate too high causing divergence, AMP fp16 overflow, or bad input data (corrupt/blank feeds).

Common situations: Raising the learning rate too aggressively; enabling mixed precision on a model sensitive to fp16; training on datasets containing misaligned/empty faces; resuming from a bad state.

Related errors

AI-assisted analysis of deepfakes/faceswap@f530cb7508 (2026-08-15). Data as JSON: /api/errors/8d8e5ca5fbf202a6. Report an issue: GitHub.