{"record":{"id":"1edbb46801b3209f","repo":"unslothai/unsloth","slug":"str-exc-1edbb4","errorCode":null,"errorMessage":"str(exc)","messagePattern":"str\\(exc\\)","errorType":"http","errorClass":"HTTPException","httpStatus":409,"severity":"warning","filePath":"studio/backend/routes/training.py","lineNumber":2535,"sourceCode":"    The check above only covers the instant it runs: every one of these endpoints then hands its\n    filesystem work to a thread, and a ``/diffusion/start`` reserving in that gap would move\n    captions or images underneath the preflight or the running trainer. As a yield dependency the\n    registration spans the endpoint, so ``reserve()`` sees it and refuses instead. Fails open on an\n    import error, like the check it replaces.\"\"\"\n    try:\n        from core.training.diffusion_training_service import (\n            TrainingActiveError,\n            get_diffusion_training_service,\n        )\n        service = get_diffusion_training_service()\n    except Exception:  # noqa: BLE001 -- unknowable state never blocks a mutation\n        yield\n        return\n    try:\n        with service.dataset_mutation():\n            yield\n    except TrainingActiveError as exc:\n        raise HTTPException(status_code = 409, detail = str(exc)) from exc\n\n\ndef _free_gpu_for_diffusion_training() -> None:\n    \"\"\"Free GPU residents before the diffusion trainer spawns its own SDXL pipeline.\n\n    The trainer subprocess loads a full SDXL pipeline; an export worker, a resident\n    Images pipeline, or loaded chat models would otherwise keep their VRAM allocated and\n    OOM the run. Mirrors the LLM start path's pre-spawn cleanup (export + diffusion\n    pipeline + chat). Best-effort: a failure to free one resident never blocks the start.\"\"\"\n    try:\n        from core.export import get_export_backend\n        exp_backend = get_export_backend()\n        if exp_backend.current_checkpoint or exp_backend.is_export_active():\n            logger.info(\"Shutting down export subprocess to free GPU memory for diffusion training\")\n            exp_backend._shutdown_subprocess()\n            exp_backend.current_checkpoint = None\n            exp_backend.is_vision = False\n            exp_backend.is_peft = False","sourceCodeStart":2517,"sourceCodeEnd":2553,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/routes/training.py#L2517-L2553","documentation":"HTTP 409 with the TrainingActiveError text, raised by the diffusion_dataset_interlock dependency: the mutation endpoint runs inside service.dataset_mutation(), a context manager the diffusion training service uses to serialize dataset access. If the service reports a run active at lock acquisition, TrainingActiveError escapes the with-block and is converted to a 409 here. Unlike the instant check (error 1311), this holds for the whole request, closing the race where training starts mid-mutation.","triggerScenarios":"Any image dataset mutation request that passes the instant _require_diffusion_dataset_mutable check but then blocks on service.dataset_mutation() while a diffusion run holds the dataset; or a run starting between the two checks.","commonSituations":"A mutation request racing a training start; large uploads whose request duration overlaps a training start triggered from another client.","solutions":["Wait for the active diffusion run to stop, then retry the mutation.","Check training status before uploading/importing large batches to minimize the race window.","Batch dataset edits into fewer, larger requests done while training is idle."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"async function mutateDatasetSafe(fn, attempts = 3) {\n  for (;;) {\n    try { return await fn() }\n    catch (e) {\n      if (e.status === 409 && /training/i.test(e.detail) && --attempts > 0) { await sleep(5_000); continue }\n      throw e\n    }\n  }\n}","preventionTips":["Check diffusion training status before large uploads/imports to shrink the race window.","Do fewer, larger mutation batches while training is idle instead of many small ones during a run.","Remember this 409 can fire even after an instant pre-check passed — a run may have started in between."],"tags":["diffusion","dataset","interlock","lock","http-409"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}