{"record":{"id":"4dfd3e847bc1fe37","repo":"unslothai/unsloth","slug":"resume-checkpoint-must-include-saved-trainer-state","errorCode":null,"errorMessage":"Resume checkpoint must include saved trainer state.","messagePattern":"Resume checkpoint must include saved trainer state\\.","errorType":"http","errorClass":"HTTPException","httpStatus":400,"severity":"error","filePath":"studio/backend/routes/training.py","lineNumber":1293,"sourceCode":"                if resume_run and await asyncio.to_thread(\n                    has_resume_state, resume_run.get(\"output_dir\")\n                ):\n                    from core.training.provenance import (\n                        resource_provenance_resume_blocker,\n                    )\n                    blocker = await asyncio.to_thread(\n                        resource_provenance_resume_blocker,\n                        training_run_config(resume_run),\n                    )\n                    if blocker:\n                        detail = blocker\n                raise HTTPException(status_code = 400, detail = detail)\n            resume_checkpoint = await asyncio.to_thread(\n                get_resume_checkpoint_path,\n                resume_output_dir,\n            )\n            if not resume_checkpoint:\n                raise HTTPException(\n                    status_code = 400,\n                    detail = \"Resume checkpoint must include saved trainer state.\",\n                )\n            request.resume_from_checkpoint = resume_checkpoint\n            (\n                resume_actual_model_repo_id,\n                resume_requires_exact_model,\n                resume_requires_exact_dataset,\n                resume_requires_exact_resources,\n                resume_model_load_mode,\n            ) = await asyncio.to_thread(\n                _prepare_resume_resource_provenance,\n                request,\n                resume_run,\n            )\n\n        if request.local_datasets:\n            request.local_datasets = _validate_local_dataset_paths(","sourceCodeStart":1275,"sourceCodeEnd":1311,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/routes/training.py#L1275-L1311","documentation":"HTTP 400 when get_resume_checkpoint_path finds no resumable checkpoint inside the (otherwise valid) resume output dir - the directory exists and maps to a run, but contains no saved trainer state to resume from.","triggerScenarios":"Resume request whose output dir passed run checks but contains no checkpoint with trainer state (e.g., training saved logs/model only, or checkpoints were deleted to save disk).","commonSituations":"Checkpoint cleanup scripts removed checkpoint-* dirs; run interrupted at step 0 before first save; user pointed at an eval or export directory.","solutions":["Inspect the output dir for checkpoint-* folders containing trainer_state files; if absent, resume is impossible","If checkpoints were deleted, restart training from the base model instead of resuming","For future runs, configure save_steps/save_strategy so checkpoints are written before long training phases"],"exampleFix":"# before\nls /data/runs/my-run/output  # only logs + final model, no checkpoint-*\nPOST /training/start {\"resume_from_checkpoint\": \"/data/runs/my-run/output\"}  # 400\n\n# after\nPOST /training/start {\"resume_from_checkpoint\": \"/data/runs/complete-run/output\"}  # has checkpoint-500/","handlingStrategy":"validation","validationCode":"from pathlib import Path\n\ndef has_saved_trainer_state(output_dir: str) -> bool:\n    d = Path(output_dir)\n    return any((c / \"trainer_state.json\").is_file() for c in d.glob(\"checkpoint-*\"))","typeGuard":null,"tryCatchPattern":"resp = client.post(\"/training/start\", payload)\nif resp.status_code == 400 and \"saved trainer state\" in resp.text:\n    # no resumable checkpoint in the dir: restart from base model instead\n    payload.pop(\"resume_from_checkpoint\")\n    resp = client.post(\"/training/start\", payload)","preventionTips":["Set save_steps/save_strategy so checkpoints exist before long training phases","Do not delete checkpoint-* directories for runs you may want to resume","Pre-check the output dir for trainer_state.json before sending the resume request"],"tags":["training","resume","checkpoint","missing-state","http-400"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}