{"record":{"id":"8530cb7ad78e1709","repo":"ultralytics/ultralytics","slug":"checkpoint-is-corrupted-with-nan-inf-weights","errorCode":null,"errorMessage":"Checkpoint {} is corrupted with NaN/Inf weights","messagePattern":"Checkpoint (.+?) is corrupted with NaN/Inf weights","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"ultralytics/engine/trainer.py","lineNumber":1046,"sourceCode":"            dist.broadcast_object_list(broadcast_list, 0)\n            corrupted = broadcast_list[0]\n        if not corrupted:\n            return False\n        if epoch == self.start_epoch:\n            LOGGER.warning(f\"{reason} detected but can not recover from last.pt...\")\n            return False  # Cannot recover on first epoch, let training continue\n        if not self.last.exists():\n            raise RuntimeError(f\"{reason} detected but no valid last.pt is available for recovery\")\n        self.nan_recovery_attempts += 1\n        if self.nan_recovery_attempts > 3:\n            raise RuntimeError(f\"Training failed: NaN persisted for {self.nan_recovery_attempts} epochs\")\n        LOGGER.warning(f\"{reason} detected (attempt {self.nan_recovery_attempts}/3), recovering from last.pt...\")\n        self._model_train()  # set model to train mode before loading checkpoint to avoid inference tensor errors\n        _, ckpt = load_checkpoint(self.last)\n        ema = ckpt[\"ema\"].float()\n        ema_state = ema.state_dict()\n        if not all(torch.isfinite(v).all() for v in ema_state.values() if isinstance(v, torch.Tensor)):\n            raise RuntimeError(f\"Checkpoint {self.last} is corrupted with NaN/Inf weights\")\n        model = unwrap_model(self.model)\n        if hasattr(model, \"student_model\"):\n            # Distillation: the EMA is stripped of the teacher (rebuilt from the distill_model path), so only the\n            # student and projector are restored; loading them separately keeps a strict key match.\n            model.student_model.load_state_dict(ema.student_model.state_dict())\n            model.projector.load_state_dict(ema.projector.state_dict())\n        else:\n            model.load_state_dict(ema_state)  # Load EMA weights into model\n        self._load_checkpoint_state(ckpt)  # Load optimizer/scaler/EMA/best_fitness\n        del ckpt, ema, ema_state\n        self.scheduler.last_epoch = epoch - 1\n        return True\n\n    def resume_training(self, ckpt):\n        \"\"\"Resume YOLO training from a given checkpoint.\"\"\"\n        if ckpt is None or not self.resume:\n            return\n        start_epoch = ckpt.get(\"epoch\", -1) + 1","sourceCodeStart":1028,"sourceCodeEnd":1064,"githubUrl":"https://github.com/ultralytics/ultralytics/blob/0449ea011cfd6c9a0d50a0bf1043aca5190cd476/ultralytics/engine/trainer.py#L1028-L1064","documentation":"Raised by BaseTrainer's NaN-recovery path: after detecting NaN/Inf in training and successfully loading last.pt, the EMA weights restored from the checkpoint themselves contain NaN/Inf values (verified with torch.isfinite over every tensor in ema.state_dict()). The recovery mechanism refuses to restore corrupted weights, because reloading NaN would immediately re-corrupt the run, so it fails loudly instead.","triggerScenarios":"NaN appeared in a previous epoch and was saved into last.pt's EMA copy before the NaN was detected; on the next NaN detection, recovery loads last.pt, finds the EMA already non-finite, and raises. Typical when checkpointing happens after the corruption or the NaN check ran after the save.","commonSituations":"NaN erupting mid-epoch after the periodic save already wrote poisoned weights; repeated NaN epochs where each save captures increasingly corrupt state; manual checkpoint files from a previously diverged run being used as last.pt.","solutions":["Restart from best.pt or the original pretrained weights instead of last.pt (best is usually from before divergence).","Re-run with the divergence fixed: lower lr0, disable AMP, or clean bad data so the saved EMA never goes non-finite.","If best.pt is also corrupt, retrain from the base pretrained model.","Keep periodic external backups of checkpoints during long runs so a pre-NaN state always exists."],"exampleFix":"# before\nmodel = YOLO(\"runs/detect/train/weights/last.pt\")\nmodel.train(resume=True)  # Checkpoint last.pt is corrupted with NaN/Inf weights\n\n# after\nmodel = YOLO(\"runs/detect/train/weights/best.pt\")\nmodel.train(data=\"d.yaml\", lr0=0.005, epochs=100)  # fresh run from clean weights","handlingStrategy":"fallback","validationCode":"import torch\nfrom pathlib import Path\n\ndef finite_ckpt(p):\n    ck = torch.load(p, map_location=\"cpu\")\n    sd = ck[\"ema\"].float().state_dict() if ck.get(\"ema\") else ck[\"model\"].float().state_dict()\n    return all(torch.isfinite(v).all() for v in sd.values() if isinstance(v, torch.Tensor))\n\nckpt = next(p for p in [Path(\"best.pt\"), Path(\"last.pt\")] if p.is_file() and finite_ckpt(p))","typeGuard":null,"tryCatchPattern":"try:\n    model.train(resume=True)\nexcept RuntimeError as e:\n    if \"corrupted with NaN/Inf weights\" in str(e):\n        YOLO(\"best.pt\").train(data=\"d.yaml\", lr0=0.005)  # fall back to clean weights, new run\n    else:\n        raise","preventionTips":["Back up best.pt (and periodic snapshots) externally during long runs.","Address divergence immediately on first NaN so poisoned state never gets saved.","Before resuming, scan checkpoint tensors with torch.isfinite to verify integrity."],"tags":["nan","checkpoint-corruption","ema","training"],"backgroundTag":null,"analyzedSha":"0449ea011cfd6c9a0d50a0bf1043aca5190cd476","analyzedAt":"2026-08-15T02:34:13.413Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}