unslothai/unsloth · error · HTTPException

Resume checkpoint must include saved trainer state.

Error message

Resume checkpoint must include saved trainer state.

What it means

HTTP 400 when get_resume_checkpoint_path finds no resumable checkpoint inside the (otherwise valid) resume output dir - the directory exists and maps to a run, but contains no saved trainer state to resume from.

Source

Thrown at studio/backend/routes/training.py:1293

                if resume_run and await asyncio.to_thread(
                    has_resume_state, resume_run.get("output_dir")
                ):
                    from core.training.provenance import (
                        resource_provenance_resume_blocker,
                    )
                    blocker = await asyncio.to_thread(
                        resource_provenance_resume_blocker,
                        training_run_config(resume_run),
                    )
                    if blocker:
                        detail = blocker
                raise HTTPException(status_code = 400, detail = detail)
            resume_checkpoint = await asyncio.to_thread(
                get_resume_checkpoint_path,
                resume_output_dir,
            )
            if not resume_checkpoint:
                raise HTTPException(
                    status_code = 400,
                    detail = "Resume checkpoint must include saved trainer state.",
                )
            request.resume_from_checkpoint = resume_checkpoint
            (
                resume_actual_model_repo_id,
                resume_requires_exact_model,
                resume_requires_exact_dataset,
                resume_requires_exact_resources,
                resume_model_load_mode,
            ) = await asyncio.to_thread(
                _prepare_resume_resource_provenance,
                request,
                resume_run,
            )

        if request.local_datasets:
            request.local_datasets = _validate_local_dataset_paths(

View on GitHub (pinned to 203007d190)

Solutions

1. Inspect the output dir for checkpoint-* folders containing trainer_state files; if absent, resume is impossible
2. If checkpoints were deleted, restart training from the base model instead of resuming
3. For future runs, configure save_steps/save_strategy so checkpoints are written before long training phases

Example fix

# before
ls /data/runs/my-run/output  # only logs + final model, no checkpoint-*
POST /training/start {"resume_from_checkpoint": "/data/runs/my-run/output"}  # 400

# after
POST /training/start {"resume_from_checkpoint": "/data/runs/complete-run/output"}  # has checkpoint-500/

from pathlib import Path

def has_saved_trainer_state(output_dir: str) -> bool:
    d = Path(output_dir)
    return any((c / "trainer_state.json").is_file() for c in d.glob("checkpoint-*"))

resp = client.post("/training/start", payload)
if resp.status_code == 400 and "saved trainer state" in resp.text:
    # no resumable checkpoint in the dir: restart from base model instead
    payload.pop("resume_from_checkpoint")
    resp = client.post("/training/start", payload)

When it happens

Trigger: Resume request whose output dir passed run checks but contains no checkpoint with trainer state (e.g., training saved logs/model only, or checkpoints were deleted to save disk).

Common situations: Checkpoint cleanup scripts removed checkpoint-* dirs; run interrupted at step 0 before first save; user pointed at an eval or export directory.

Related errors

• Resume checkpoint must belong to a stopped or errored run wi
• The training checkpoint at '{directory}' could not be read;
• Checkpoint tensor '{name}' has shape {tuple(saved.shape)} bu
• This run has {len(trainable)} trainable tensors and the chec
• This checkpoint does not record which optimizer wrote its st

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/4dfd3e847bc1fe37. Report an issue: GitHub.