{"record":{"id":"f1b38e0995ff622e","repo":"unslothai/unsloth","slug":"resume-checkpoint-must-belong-to-a-stopped-or-erro","errorCode":null,"errorMessage":"Resume checkpoint must belong to a stopped or errored run with complete saved trainer state.","messagePattern":"Resume checkpoint must belong to a stopped or errored run with complete saved trainer state\\.","errorType":"http","errorClass":"HTTPException","httpStatus":400,"severity":"error","filePath":"studio/backend/routes/training.py","lineNumber":1287,"sourceCode":"            if not resume_run or not await asyncio.to_thread(can_resume_run, resume_run):\n                detail = \"Resume checkpoint must belong to a stopped or errored run with complete saved trainer state.\"\n                # Only when the checkpoint itself is intact. can_resume_run refuses for several reasons and\n                # the blocker is computed independently of which one fired, so asking unconditionally would\n                # answer a provenance sentence even when the checkpoint is what is missing. has_resume_state\n                # is the discriminator can_resume_run itself short-circuits on.\n                if resume_run and await asyncio.to_thread(\n                    has_resume_state, resume_run.get(\"output_dir\")\n                ):\n                    from core.training.provenance import (\n                        resource_provenance_resume_blocker,\n                    )\n                    blocker = await asyncio.to_thread(\n                        resource_provenance_resume_blocker,\n                        training_run_config(resume_run),\n                    )\n                    if blocker:\n                        detail = blocker\n                raise HTTPException(status_code = 400, detail = detail)\n            resume_checkpoint = await asyncio.to_thread(\n                get_resume_checkpoint_path,\n                resume_output_dir,\n            )\n            if not resume_checkpoint:\n                raise HTTPException(\n                    status_code = 400,\n                    detail = \"Resume checkpoint must include saved trainer state.\",\n                )\n            request.resume_from_checkpoint = resume_checkpoint\n            (\n                resume_actual_model_repo_id,\n                resume_requires_exact_model,\n                resume_requires_exact_dataset,\n                resume_requires_exact_resources,\n                resume_model_load_mode,\n            ) = await asyncio.to_thread(\n                _prepare_resume_resource_provenance,","sourceCodeStart":1269,"sourceCodeEnd":1305,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/routes/training.py#L1269-L1305","documentation":"HTTP 400 when the resume output dir does not map to a resumable run: either get_resumable_run_by_output_dir finds no run, or can_resume_run refuses (run not stopped/errored, or saved trainer state incomplete). When the checkpoint state itself exists, the server substitutes the more specific resource-provenance blocker as the detail.","triggerScenarios":"Resuming from an output dir whose run is still running (or was never recorded), or a stopped/errored run whose saved trainer state is incomplete.","commonSituations":"Resume attempted while the original job is still active; run crashed before saving complete state; output dir from a different backend instance the current one has no record of.","solutions":["Confirm the source run is in stopped or errored state (not running) before resuming","Check the run's output dir contains complete saved trainer state (trainer_state.json and checkpoints)","If state is incomplete, re-run training from scratch - a partial checkpoint cannot be resumed","Read the returned detail: if it is a provenance blocker (resource mismatch), align GPU/CPU/model resources with the original run"],"exampleFix":"// before\nPOST /training/start {\"resume_from_checkpoint\": \"/data/runs/running-job/output\"}  // 400\n\n// after\n# stop the job first, confirm state == stopped/errored, then\nPOST /training/start {\"resume_from_checkpoint\": \"/data/runs/stopped-job/output\"}","handlingStrategy":"validation","validationCode":"# before resuming, check the run's terminal state via the runs/jobs API\nrun = client.get(f\"/training/runs/by-output-dir?path={dir}\").json()\nassert run and run[\"state\"] in (\"stopped\", \"errored\"), \"run must be stopped or errored to resume\"","typeGuard":null,"tryCatchPattern":"resp = client.post(\"/training/start\", payload)\nif resp.status_code == 400 and \"stopped or errored run\" in resp.text:\n    detail = resp.json()[\"detail\"]\n    # detail may be a provenance blocker naming the mismatched resource; fix that resource","preventionTips":["Only resume runs in stopped/errored state with complete saved trainer state","Keep hardware/model resources consistent between original and resume runs","Read the detail carefully: it discriminates missing state vs provenance mismatch"],"tags":["training","resume","checkpoint","http-400","run-state"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}