unslothai/unsloth · warning · HTTPException

An LLM training job is already running. Stop it before start

Error message

An LLM training job is already running. Stop it before starting diffusion (Images) training.

What it means

HTTP 409 on the diffusion training start endpoint: the LLM training backend reports an active run (get_training_backend().is_training_active()). Both trainers claim the whole GPU, so starting diffusion training while an LLM LoRA run holds VRAM would OOM; the check is symmetric with the diffusion check inside start_training. A backend import/health failure is swallowed (fail-open) so it never blocks a start.

Source

Thrown at studio/backend/routes/training.py:2717

    if via_api_key is True:
        from core.inference.llama_keepwarm import other_inference_request_count
        if (
            other_inference_request_count(current_request_counted = False) > 0
            or _background_video_generation_active()
        ):
            raise HTTPException(
                status_code = 409,
                detail = (
                    "Cannot start diffusion (Images) training over the API while an inference "
                    "request is in progress. Wait for it to finish, or start training from the "
                    "Studio UI."
                ),
            )

    # Interlock: refuse while an LLM training run holds the GPU (symmetric with the diffusion check in start_training), so the two trainers never contend for VRAM.
    try:
        if get_training_backend().is_training_active():
            raise HTTPException(
                status_code = 409,
                detail = (
                    "An LLM training job is already running. "
                    "Stop it before starting diffusion (Images) training."
                ),
            )
    except HTTPException:
        raise
    except Exception:  # noqa: BLE001 -- backend import/health issue must not block a start
        pass

    # Resolve + contain the dataset and output paths BEFORE spawning: the trainer subprocess would otherwise resolve them relative to its own cwd.
    config = body.model_dump()
    try:
        from utils.paths import outputs_root, resolve_output_dir

        config["data_dir"] = str(_resolve_diffusion_data_dir(config["data_dir"]))
        # A name that cleans away to nothing ("." / "outputs" / "./.") resolves to the outputs ROOT, where the trainer would write the adapter flat and the is_dir()-filtered listings could never see it.

View on GitHub (pinned to 203007d190)

Solutions

1. Stop the LLM training run (POST /training/stop) and wait for it to fully release the GPU.
2. Poll /training/status until no LLM run is active, then retry the diffusion start.
3. Serialize training jobs in your workflow instead of running both trainers concurrently.

const llmStatus = await get('/training/status')
if (llmStatus.is_active) throw new Error('Stop the LLM run before starting diffusion training')
await startDiffusionTraining(payload)

try { await startDiffusionTraining(payload) } catch (e) { if (e.status === 409 && /LLM training job is already running/.test(e.detail)) { await post('/training/stop', {save: true}); await waitUntilIdle(); return startDiffusionTraining(payload) } throw e }

When it happens

Trigger: POST diffusion training start while an LLM (text) LoRA training job is running in the same Studio instance.

Common situations: Chaining text-then-image finetunes without waiting for the first to finish; a long LLM run left overnight while the user starts an Images run in the morning.

Related errors

• Cannot start diffusion (Images) training over the API while
• Training images cannot be changed while diffusion training i
• str(exc)
• transformer_quant='{requested}' could not be used: {reason}.
• {str(exc)}

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/08079c2cebd641dc. Report an issue: GitHub.