{"record":{"id":"789d21b9f800015b","repo":"unslothai/unsloth","slug":"resume-from-checkpoint-is-not-supported-for-resol","errorCode":null,"errorMessage":"resume_from_checkpoint is not supported for {resolved_family}: its trainer writes no checkpoint bundle, so there is nothing to continue from and the run would silently start over and overwrite its output. Start a fresh run.","messagePattern":"resume_from_checkpoint is not supported for (.+?): its trainer writes no checkpoint bundle, so there is nothing to continue from and the run would silently start over and overwrite its output\\. Start a fresh run\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"studio/backend/core/training/diffusion_train_common.py","lineNumber":1078,"sourceCode":"            ) from exc\n        if save_steps < 0:\n            raise ValueError(\"save_steps must be >= 0 (0 disables periodic checkpoints)\")\n        if save_total_limit < 0:\n            raise ValueError(\"save_total_limit must be >= 0 (0 keeps every checkpoint)\")\n        # A blank resume path (the Studio default when the field is present but unset) means \"fresh run\", not the outputs root.\n        resume_from_checkpoint = (\n            str(self.resume_from_checkpoint).strip()\n            if self.resume_from_checkpoint is not None\n            else \"\"\n        ) or None\n        # The H3 loop does not checkpoint: it neither writes a resume bundle nor restores one.\n        # Accepting these two silently was the dangerous part -- a caller handing over a resume\n        # bundle got a FRESH optimization that then overwrote the outputs it was meant to\n        # continue, and one asking for periodic saves got none, both discovered only after an\n        # expensive run. Refuse in validation, where it costs nothing, until the loop supports it.\n        if resolved_family in CHECKPOINTLESS_FAMILIES:\n            if resume_from_checkpoint:\n                raise ValueError(\n                    f\"resume_from_checkpoint is not supported for {resolved_family}: its trainer \"\n                    f\"writes no checkpoint bundle, so there is nothing to continue from and the \"\n                    f\"run would silently start over and overwrite its output. Start a fresh run.\"\n                )\n            if save_steps:\n                raise ValueError(\n                    f\"save_steps is not supported for {resolved_family}: its trainer writes no \"\n                    f\"checkpoint bundle. Leave it at 0; the adapter is still saved at the end.\"\n                )\n        try:\n            ema_decay = float(self.ema_decay or 0.0)\n        except (TypeError, ValueError) as exc:\n            raise ValueError(f\"ema_decay must be a number, got {self.ema_decay!r}\") from exc\n        # decay = 1.0 would freeze the shadow at its init forever; the update is shadow * decay + param * (1 - decay), so valid decays live in [0, 1).\n        if not 0.0 <= ema_decay < 1.0:\n            raise ValueError(\"ema_decay must be in [0, 1); 0 disables the EMA adapter\")\n        # A blank cond_cache_dir (the Studio default when unset) means \"off\", not cwd.\n        cond_cache_dir = (","sourceCodeStart":1060,"sourceCodeEnd":1096,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/training/diffusion_train_common.py#L1060-L1096","documentation":"Raised when resume_from_checkpoint is set for a family in CHECKPOINTLESS_FAMILIES (the H3 loop). That trainer writes no checkpoint bundle at all, so there is literally nothing to resume from — previously this was accepted silently and the 'resume' started a FRESH optimization that overwrote the very outputs it was meant to continue. Validation now refuses it where it costs nothing.","triggerScenarios":"Submitting a resume request for a checkpointless family: resume_from_checkpoint pointing at any path (a blank/whitespace-only path is treated as fresh-run and allowed; only a non-empty path triggers this). Typically a generic resume workflow applied uniformly across families.","commonSituations":"An orchestration layer that auto-resumes interrupted jobs by family-agnostic logic; users clicking 'resume' in a UI that shows the option for every family; crash-recovery scripts that always pass the last output dir.","solutions":["Start a fresh run for this family — its trainer has no resume capability.","If you need resumable long runs, pick a family whose trainer writes checkpoint bundles (not in CHECKPOINTLESS_FAMILIES).","Gate any generic resume automation on family capability: skip/flag checkpointless families instead of submitting."],"exampleFix":"# before\nconfig = TrainConfig(base_model='h3-family-model', resume_from_checkpoint='outputs/run-42')\n\n# after\nconfig = TrainConfig(base_model='h3-family-model')  # fresh run; H3 cannot resume","handlingStrategy":"validation","validationCode":"CHECKPOINTLESS_FAMILIES = {...}  # mirror the trainer's set\n\ndef check_resume(family, resume_from_checkpoint) -> None:\n    path = str(resume_from_checkpoint).strip() if resume_from_checkpoint is not None else \"\"\n    if family in CHECKPOINTLESS_FAMILIES and path:\n        raise ValueError(\n            f\"family {family} writes no checkpoint bundle; resume would silently \"\n            f\"overwrite outputs — start a fresh run\"\n        )","typeGuard":"def family_supports_resume(family) -> bool:\n    return family not in CHECKPOINTLESS_FAMILIES","tryCatchPattern":"try:\n    session.submit_training(config)\nexcept ValueError as e:\n    if \"resume_from_checkpoint is not supported\" in str(e):\n        config.resume_from_checkpoint = None  # degrade to a fresh run, do NOT blind-retry\n        session.submit_training(config)\n    else:\n        raise","preventionTips":["Never auto-resume by family-agnostic logic — gate on a family->capability map first.","Treat 'resume accepted silently' as a red flag: this error exists precisely because silent acceptance destroyed outputs.","Hide/disable resume UI affordances for checkpointless families."],"tags":["training","resume","checkpointing","validation","data-loss"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}