unslothai/unsloth · warning · HTTPException

str(exc)

Error message

str(exc)

What it means

HTTP 409 with the TrainingActiveError text, raised by the diffusion_dataset_interlock dependency: the mutation endpoint runs inside service.dataset_mutation(), a context manager the diffusion training service uses to serialize dataset access. If the service reports a run active at lock acquisition, TrainingActiveError escapes the with-block and is converted to a 409 here. Unlike the instant check (error 1311), this holds for the whole request, closing the race where training starts mid-mutation.

Source

Thrown at studio/backend/routes/training.py:2535

    The check above only covers the instant it runs: every one of these endpoints then hands its
    filesystem work to a thread, and a ``/diffusion/start`` reserving in that gap would move
    captions or images underneath the preflight or the running trainer. As a yield dependency the
    registration spans the endpoint, so ``reserve()`` sees it and refuses instead. Fails open on an
    import error, like the check it replaces."""
    try:
        from core.training.diffusion_training_service import (
            TrainingActiveError,
            get_diffusion_training_service,
        )
        service = get_diffusion_training_service()
    except Exception:  # noqa: BLE001 -- unknowable state never blocks a mutation
        yield
        return
    try:
        with service.dataset_mutation():
            yield
    except TrainingActiveError as exc:
        raise HTTPException(status_code = 409, detail = str(exc)) from exc


def _free_gpu_for_diffusion_training() -> None:
    """Free GPU residents before the diffusion trainer spawns its own SDXL pipeline.

    The trainer subprocess loads a full SDXL pipeline; an export worker, a resident
    Images pipeline, or loaded chat models would otherwise keep their VRAM allocated and
    OOM the run. Mirrors the LLM start path's pre-spawn cleanup (export + diffusion
    pipeline + chat). Best-effort: a failure to free one resident never blocks the start."""
    try:
        from core.export import get_export_backend
        exp_backend = get_export_backend()
        if exp_backend.current_checkpoint or exp_backend.is_export_active():
            logger.info("Shutting down export subprocess to free GPU memory for diffusion training")
            exp_backend._shutdown_subprocess()
            exp_backend.current_checkpoint = None
            exp_backend.is_vision = False
            exp_backend.is_peft = False

View on GitHub (pinned to 203007d190)

Solutions

1. Wait for the active diffusion run to stop, then retry the mutation.
2. Check training status before uploading/importing large batches to minimize the race window.
3. Batch dataset edits into fewer, larger requests done while training is idle.

async function mutateDatasetSafe(fn, attempts = 3) {
  for (;;) {
    try { return await fn() }
    catch (e) {
      if (e.status === 409 && /training/i.test(e.detail) && --attempts > 0) { await sleep(5_000); continue }
      throw e
    }
  }
}

When it happens

Trigger: Any image dataset mutation request that passes the instant _require_diffusion_dataset_mutable check but then blocks on service.dataset_mutation() while a diffusion run holds the dataset; or a run starting between the two checks.

Common situations: A mutation request racing a training start; large uploads whose request duration overlaps a training start triggered from another client.

Related errors

• Training images cannot be changed while diffusion training i
• An LLM training job is already running. Stop it before start
• Cannot start diffusion (Images) training over the API while
• Unknown model_kind '{model_kind}'. Expected one of {sorted(_
• Invalid base64 image data: {exc}

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/1edbb46801b3209f. Report an issue: GitHub.