unslothai/unsloth · warning · HTTPException

Training is still running. Stop training and wait for it to

Error message

Training is still running. Stop training and wait for it to finish before resetting.

What it means

HTTP 409 from the reset endpoint when backend.reset_training_state returns 'active': training is currently running and the backend refuses to reset training state (which would discard progress/configuration) underneath a live run. The user must stop training and let it fully finish before resetting.

Source

Thrown at studio/backend/routes/training.py:1808

            log = logger,
        )


@router.post("/reset")
async def reset_training(
    body: Optional[TrainingResetRequest] = None, current_subject: str = Depends(get_current_subject)
):
    """Reset training state so the user can return to configuration."""
    try:
        backend = get_training_backend()
        result = await asyncio.to_thread(
            backend.reset_training_state,
            expected_job_id = body.expected_job_id if body is not None else None,
        )
        if result == "superseded":
            return {"status": "superseded"}
        if result == "active":
            raise HTTPException(
                status_code = 409,
                detail = "Training is still running. Stop training and wait for it to finish before resetting.",
            )
        return {"status": "ok"}
    except HTTPException:
        raise
    except Exception as e:
        raise log_and_http_error(
            e,
            500,
            "Failed to reset training",
            event = "training.reset_failed",
            log = logger,
        )


def _training_status_identity(backend) -> TrainingStatusIdentitySnapshot:
    snapshot = getattr(backend, "training_status_identity", None)

View on GitHub (pinned to 203007d190)

Solutions

1. Stop training first (POST /training/stop) and poll /training/status until it reports the run is finished/idle.
2. Then retry the reset request.
3. If status stays active unusually long, check the training log for a hung subprocess before forcing anything.

Example fix

// before
await post('/training/stop', {save: true})
await post('/training/reset')  // 409: still running
// after
await post('/training/stop', {save: true})
while ((await get('/training/status')).is_active) await sleep(2000)
await post('/training/reset')

const status = await get('/training/status')
if (status.is_active) throw new Error('Stop training and wait for it to finish before resetting')
await post('/training/reset', body)

try { await post('/training/reset', body) } catch (e) { if (e.status === 409 && /still running/.test(e.detail)) { await post('/training/stop', {save: true}); await waitUntilIdle(); return post('/training/reset', body) } throw e }

When it happens

Trigger: POST /training/reset while a training subprocess is active — including the window where a stop was requested but the run has not yet halted at its next safe step.

Common situations: Clicking 'reset' right after clicking 'stop' without waiting for the stop to take effect; a background job whose shutdown takes seconds to minutes for large models.

Related errors

• {str(exc)}
• Cannot start diffusion (Images) training over the API while
• An LLM training job is already running. Stop it before start
• No export subprocess running
• No diffusion model is loaded.

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/afc1f9a9c944dc54. Report an issue: GitHub.