unslothai/unsloth · error · HTTPException

{validation_message}

Error message

{validation_message}

What it means

HTTP 400 whose detail is the raw ValueError message from normalize_resume_output_dir: the resume_from_checkpoint value could not be normalized to a valid run output directory (bad path shape, non-run directory, unsafe characters, or similar path validation failure).

Source

Thrown at studio/backend/routes/training.py:1263

            )

        resume_output_dir: Optional[str] = None
        resume_run: Optional[dict] = None
        resume_actual_model_repo_id: Optional[str] = None
        resume_model_load_mode: Optional[str] = None
        resume_requires_exact_resources = False
        resume_requires_exact_model = False
        resume_requires_exact_dataset = False
        if request.resume_from_checkpoint:
            try:
                resume_output_dir = await asyncio.to_thread(
                    normalize_resume_output_dir,
                    request.resume_from_checkpoint,
                )
            except ValueError as e:
                # Deliberate user-facing validation message.
                validation_message = str(e)
                raise HTTPException(status_code = 400, detail = validation_message)

            resume_run = await asyncio.to_thread(
                get_resumable_run_by_output_dir,
                resume_output_dir,
            )
            if not resume_run or not await asyncio.to_thread(can_resume_run, resume_run):
                detail = "Resume checkpoint must belong to a stopped or errored run with complete saved trainer state."
                # Only when the checkpoint itself is intact. can_resume_run refuses for several reasons and
                # the blocker is computed independently of which one fired, so asking unconditionally would
                # answer a provenance sentence even when the checkpoint is what is missing. has_resume_state
                # is the discriminator can_resume_run itself short-circuits on.
                if resume_run and await asyncio.to_thread(
                    has_resume_state, resume_run.get("output_dir")
                ):
                    from core.training.provenance import (
                        resource_provenance_resume_blocker,
                    )
                    blocker = await asyncio.to_thread(

View on GitHub (pinned to 203007d190)

Solutions

1. Read the returned validation_message - it states exactly why the path was rejected
2. Pass the run's top-level output directory (the one containing trainer state), not a checkpoint subfolder or model dir
3. Verify the path exists and was produced by a previous training run of this backend
4. Use absolute paths to avoid cwd-relative resolution surprises

Example fix

// before
POST /training/start {"resume_from_checkpoint": "checkpoint-500"}  // 400

// after
POST /training/start {"resume_from_checkpoint": "/data/runs/my-run/output"}

from pathlib import Path

def valid_resume_output_dir(p: str) -> bool:
    d = Path(p)
    return d.is_absolute() and d.is_dir() and any(d.glob("checkpoint-*"))

def is_resume_output_dir(v: object) -> TypeGuard[str]:
    return isinstance(v, str) and valid_resume_output_dir(v)

resp = client.post("/training/start", payload)
if resp.status_code == 400 and "resume_from_checkpoint" in payload:
    show(resp.json()["detail"])  # server's validation_message names the exact path problem

When it happens

Trigger: POST /training/start with resume_from_checkpoint that is not a resolvable training output dir: relative junk path, a file instead of a directory, path with traversal, or a directory lacking run layout.

Common situations: Passing a checkpoint subdirectory instead of the run's output dir; typos; moving run directories so the stored path no longer resolves; copy/pasting a model dir rather than the trainer output dir.

Related errors

• Resume checkpoint must belong to a stopped or errored run wi
• Resume checkpoint must include saved trainer state.
• The training checkpoint at '{directory}' could not be read;
• resume_from_checkpoint is not supported for {resolved_family
• Checkpoint tensor '{name}' has shape {tuple(saved.shape)} bu

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/02f14ec90db7c368. Report an issue: GitHub.