unslothai/unsloth · error · HTTPException

The source run contains invalid training configuration and c

Error message

The source run contains invalid training configuration and cannot be resumed safely.

What it means

Thrown (HTTP 409) when resuming a training run whose persisted configuration fails re-validation against the current TrainingStartRequest schema. The resume flow copies stored field values back onto the request and re-runs model_validate; any field that no longer satisfies the schema (type, constraint, or enum change) triggers this error. It is a safety refusal: silently resuming with an invalid config could corrupt the run.

Source

Thrown at studio/backend/routes/training.py:1008

        raise HTTPException(
            status_code = 409,
            detail = "The selected dataset does not match the dataset used by the source run.",
        )

    request.model_name = stored_model
    for field, default in _RESUME_DATASET_DEFAULTS.items():
        value = stored.get(field, default)
        setattr(request, field, list(value) if isinstance(value, list) else value)
    request.s3_config = None
    for field in _RESUME_CACHE_FIELDS:
        default = False if field.endswith("_known_cached") else None
        setattr(request, field, stored.get(field, default))
    try:
        validated_request = TrainingStartRequest.model_validate(
            {field: getattr(request, field) for field in TrainingStartRequest.model_fields}
        )
    except ValidationError as error:
        raise HTTPException(
            status_code = 409,
            detail = (
                "The source run contains invalid training configuration and cannot "
                "be resumed safely."
            ),
        ) from error
    for field in TrainingStartRequest.model_fields:
        setattr(request, field, getattr(validated_request, field))
    marker = stored.get(RESOURCE_PROVENANCE_KEY)
    model_load_mode = (
        _normalized_optional_string(marker.get("model_load_mode"))
        if requires_exact_model and isinstance(marker, dict)
        else None
    )
    return (
        _normalized_optional_string(stored.get("actual_model_repo_id")),
        requires_exact_model,
        requires_exact_dataset,

View on GitHub (pinned to 203007d190)

Solutions

1. Check the server log for the chained ValidationError ('from error') to see exactly which stored field fails validation
2. Edit or remove the offending stored config values in the source run's stored record / output dir, then retry the resume
3. If the stored config is irreconcilable with the current schema, start a fresh training run instead of resuming
4. Re-create the run with an older backend version that matches the schema the run was saved with, save a clean checkpoint, then upgrade

Example fix

// before
POST /training/start {"resume_from_checkpoint": "/runs/old-run"}  // 409

// after
# 1) server log shows: field 'learning_rate' stored as string, now requires float
# 2) fix stored value in the run record, then
POST /training/start {"resume_from_checkpoint": "/runs/old-run"}

try:
    resp = client.post("/training/start", payload)
except HTTPStatusError as e:
    if e.response.status_code == 409 and "cannot be resumed safely" in e.response.text:
        # inspect server logs for the chained ValidationError; fix stored config or start fresh
        raise ResumeConfigIncompatible(run_id=payload["resume_from_checkpoint"])
    raise

When it happens

Trigger: POST to resume training with resume_from_checkpoint pointing at a run saved by an older backend version whose stored config contains values now invalid under the current TrainingStartRequest model (renamed enums, tightened min/max constraints, changed types).

Common situations: Upgrading the studio backend after schema migrations; resuming runs created before a validation rule was added; hand-edited run config files in the output directory.

Related errors

• resume_from_checkpoint is not supported for {resolved_family
• This run has {len(trainable)} trainable tensors and the chec
• The training checkpoint at '{directory}' could not be read;
• gradient_accumulation_steps must be >= 1
• lora_rank must be >= 1

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/d928ed308283ea8a. Report an issue: GitHub.