ultralytics/ultralytics · critical · RuntimeError

Checkpoint {} is corrupted with NaN/Inf weights

Error message

Checkpoint {} is corrupted with NaN/Inf weights

What it means

Raised by BaseTrainer's NaN-recovery path: after detecting NaN/Inf in training and successfully loading last.pt, the EMA weights restored from the checkpoint themselves contain NaN/Inf values (verified with torch.isfinite over every tensor in ema.state_dict()). The recovery mechanism refuses to restore corrupted weights, because reloading NaN would immediately re-corrupt the run, so it fails loudly instead.

Source

Thrown at ultralytics/engine/trainer.py:1046

            dist.broadcast_object_list(broadcast_list, 0)
            corrupted = broadcast_list[0]
        if not corrupted:
            return False
        if epoch == self.start_epoch:
            LOGGER.warning(f"{reason} detected but can not recover from last.pt...")
            return False  # Cannot recover on first epoch, let training continue
        if not self.last.exists():
            raise RuntimeError(f"{reason} detected but no valid last.pt is available for recovery")
        self.nan_recovery_attempts += 1
        if self.nan_recovery_attempts > 3:
            raise RuntimeError(f"Training failed: NaN persisted for {self.nan_recovery_attempts} epochs")
        LOGGER.warning(f"{reason} detected (attempt {self.nan_recovery_attempts}/3), recovering from last.pt...")
        self._model_train()  # set model to train mode before loading checkpoint to avoid inference tensor errors
        _, ckpt = load_checkpoint(self.last)
        ema = ckpt["ema"].float()
        ema_state = ema.state_dict()
        if not all(torch.isfinite(v).all() for v in ema_state.values() if isinstance(v, torch.Tensor)):
            raise RuntimeError(f"Checkpoint {self.last} is corrupted with NaN/Inf weights")
        model = unwrap_model(self.model)
        if hasattr(model, "student_model"):
            # Distillation: the EMA is stripped of the teacher (rebuilt from the distill_model path), so only the
            # student and projector are restored; loading them separately keeps a strict key match.
            model.student_model.load_state_dict(ema.student_model.state_dict())
            model.projector.load_state_dict(ema.projector.state_dict())
        else:
            model.load_state_dict(ema_state)  # Load EMA weights into model
        self._load_checkpoint_state(ckpt)  # Load optimizer/scaler/EMA/best_fitness
        del ckpt, ema, ema_state
        self.scheduler.last_epoch = epoch - 1
        return True

    def resume_training(self, ckpt):
        """Resume YOLO training from a given checkpoint."""
        if ckpt is None or not self.resume:
            return
        start_epoch = ckpt.get("epoch", -1) + 1

View on GitHub (pinned to 0449ea011c)

Solutions

1. Restart from best.pt or the original pretrained weights instead of last.pt (best is usually from before divergence).
2. Re-run with the divergence fixed: lower lr0, disable AMP, or clean bad data so the saved EMA never goes non-finite.
3. If best.pt is also corrupt, retrain from the base pretrained model.
4. Keep periodic external backups of checkpoints during long runs so a pre-NaN state always exists.

Example fix

# before
model = YOLO("runs/detect/train/weights/last.pt")
model.train(resume=True)  # Checkpoint last.pt is corrupted with NaN/Inf weights

# after
model = YOLO("runs/detect/train/weights/best.pt")
model.train(data="d.yaml", lr0=0.005, epochs=100)  # fresh run from clean weights

import torch
from pathlib import Path

def finite_ckpt(p):
    ck = torch.load(p, map_location="cpu")
    sd = ck["ema"].float().state_dict() if ck.get("ema") else ck["model"].float().state_dict()
    return all(torch.isfinite(v).all() for v in sd.values() if isinstance(v, torch.Tensor))

ckpt = next(p for p in [Path("best.pt"), Path("last.pt")] if p.is_file() and finite_ckpt(p))

try:
    model.train(resume=True)
except RuntimeError as e:
    if "corrupted with NaN/Inf weights" in str(e):
        YOLO("best.pt").train(data="d.yaml", lr0=0.005)  # fall back to clean weights, new run
    else:
        raise

When it happens

Trigger: NaN appeared in a previous epoch and was saved into last.pt's EMA copy before the NaN was detected; on the next NaN detection, recovery loads last.pt, finds the EMA already non-finite, and raises. Typical when checkpointing happens after the corruption or the NaN check ran after the save.

Common situations: NaN erupting mid-epoch after the periodic save already wrote poisoned weights; repeated NaN epochs where each save captures increasingly corrupt state; manual checkpoint files from a previously diverged run being used as last.pt.

Related errors

• {} detected but no valid last.pt is available for recovery
• Training failed: NaN persisted for {} epochs
• {self.prefix}No labels found in {path}. {HELP_URL}
• Segment dataset requires equal numbers of boxes and segments
• 'single_cls=True' is not supported for semantic segmentation

AI-assisted analysis of ultralytics/ultralytics@0449ea011c (2026-08-15). Data as JSON: /api/errors/8530cb7ad78e1709. Report an issue: GitHub.