unslothai/unsloth · error · HTTPException

Resume checkpoint must belong to a stopped or errored run wi

Error message

Resume checkpoint must belong to a stopped or errored run with complete saved trainer state.

What it means

HTTP 400 when the resume output dir does not map to a resumable run: either get_resumable_run_by_output_dir finds no run, or can_resume_run refuses (run not stopped/errored, or saved trainer state incomplete). When the checkpoint state itself exists, the server substitutes the more specific resource-provenance blocker as the detail.

Source

Thrown at studio/backend/routes/training.py:1287

            if not resume_run or not await asyncio.to_thread(can_resume_run, resume_run):
                detail = "Resume checkpoint must belong to a stopped or errored run with complete saved trainer state."
                # Only when the checkpoint itself is intact. can_resume_run refuses for several reasons and
                # the blocker is computed independently of which one fired, so asking unconditionally would
                # answer a provenance sentence even when the checkpoint is what is missing. has_resume_state
                # is the discriminator can_resume_run itself short-circuits on.
                if resume_run and await asyncio.to_thread(
                    has_resume_state, resume_run.get("output_dir")
                ):
                    from core.training.provenance import (
                        resource_provenance_resume_blocker,
                    )
                    blocker = await asyncio.to_thread(
                        resource_provenance_resume_blocker,
                        training_run_config(resume_run),
                    )
                    if blocker:
                        detail = blocker
                raise HTTPException(status_code = 400, detail = detail)
            resume_checkpoint = await asyncio.to_thread(
                get_resume_checkpoint_path,
                resume_output_dir,
            )
            if not resume_checkpoint:
                raise HTTPException(
                    status_code = 400,
                    detail = "Resume checkpoint must include saved trainer state.",
                )
            request.resume_from_checkpoint = resume_checkpoint
            (
                resume_actual_model_repo_id,
                resume_requires_exact_model,
                resume_requires_exact_dataset,
                resume_requires_exact_resources,
                resume_model_load_mode,
            ) = await asyncio.to_thread(
                _prepare_resume_resource_provenance,

View on GitHub (pinned to 203007d190)

Solutions

1. Confirm the source run is in stopped or errored state (not running) before resuming
2. Check the run's output dir contains complete saved trainer state (trainer_state.json and checkpoints)
3. If state is incomplete, re-run training from scratch - a partial checkpoint cannot be resumed
4. Read the returned detail: if it is a provenance blocker (resource mismatch), align GPU/CPU/model resources with the original run

Example fix

// before
POST /training/start {"resume_from_checkpoint": "/data/runs/running-job/output"}  // 400

// after
# stop the job first, confirm state == stopped/errored, then
POST /training/start {"resume_from_checkpoint": "/data/runs/stopped-job/output"}

# before resuming, check the run's terminal state via the runs/jobs API
run = client.get(f"/training/runs/by-output-dir?path={dir}").json()
assert run and run["state"] in ("stopped", "errored"), "run must be stopped or errored to resume"

resp = client.post("/training/start", payload)
if resp.status_code == 400 and "stopped or errored run" in resp.text:
    detail = resp.json()["detail"]
    # detail may be a provenance blocker naming the mismatched resource; fix that resource

When it happens

Trigger: Resuming from an output dir whose run is still running (or was never recorded), or a stopped/errored run whose saved trainer state is incomplete.

Common situations: Resume attempted while the original job is still active; run crashed before saving complete state; output dir from a different backend instance the current one has no record of.

Related errors

• Resume checkpoint must include saved trainer state.
• The training checkpoint at '{directory}' could not be read;
• Checkpoint tensor '{name}' has shape {tuple(saved.shape)} bu
• This run has {len(trainable)} trainable tensors and the chec
• This checkpoint does not record which optimizer wrote its st

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/f1b38e0995ff622e. Report an issue: GitHub.