What causes "Training state changed during status read" in unslothai/unsloth?

Retry the GET after a short delay — the next read usually observes a stable state.

unslothai/unsloth · info · HTTPException

Training state changed during status read

Error message

Training state changed during status read

What it means

HTTP 409 from the status endpoint: it builds a consistent training-status snapshot by comparing a backend identity token before/after each read, retrying up to 3 times. If the identity (job id / spawn state) changed during every attempt — i.e. a training job started, stopped, or was replaced concurrently with each read — it gives up and returns 409 rather than a torn snapshot.

Source

Thrown at studio/backend/routes/training.py:1993


@router.get("/status")
async def get_training_status(current_subject: str = Depends(get_current_subject)):
    """
    Get the current training status.
    """
    try:
        backend = get_training_backend()
        for _ in range(3):
            identity_before = _training_status_identity(backend)
            is_active = await asyncio.to_thread(_run_active, backend)
            identity = _training_status_identity(backend)
            if identity != identity_before:
                continue
            status = _build_training_status(backend, identity, is_active)
            if _training_status_identity(backend) == identity:
                return status
        raise HTTPException(status_code = 409, detail = "Training state changed during status read")
    except HTTPException:
        raise
    except Exception as e:
        raise log_and_http_error(
            e,
            500,
            "Failed to get training status",
            event = "training.status_failed",
            log = logger,
        )


@router.get("/metrics", response_model = TrainingMetricsResponse)
async def get_training_metrics(
    expected_job_id: Optional[str] = None, current_subject: str = Depends(get_current_subject)
):
    """
    Get training metrics (loss, learning rate, steps).

View on GitHub (pinned to 203007d190)

Solutions

Retry the GET after a short delay — the next read usually observes a stable state.
Space out polling (>= 1-2s) and stop polling during explicit start/stop transitions.
Key subsequent requests off the job_id in the successful status response to detect supersession cleanly.

Example fix

// before
const status = await get('/training/status')
// after
let status
for (let i = 0; i < 5; i++) {
  try { status = await get('/training/status'); break }
  catch (e) { if (e.status !== 409) throw e; await sleep(1000) }
}

Defensive patterns

Strategy: retry

Try / catch

async function getStatusSafe() {
  for (let i = 0; i < 5; i++) {
    try { return await get('/training/status') }
    catch (e) { if (e.status === 409 && /state changed/.test(e.detail)) { await sleep(1000); continue } throw e }
  }
  throw new Error('Training status unstable after 5 attempts')
}

Prevention

Poll status at 1-2s intervals, not sub-second.
Pause polling while a start/stop/reset request is in flight.
Never build UI state from a torn snapshot — always use the returned job_id as your consistency key.

When it happens

Trigger: GET /training/status issued at the exact moment a training job starts, stops, resets, or is superseded, such that all 3 read attempts straddle a transition.

Common situations: Aggressive status polling (sub-second intervals) while a job is being started/stopped from another tab or by automation; slow reads on large histories making the race window wider.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/d7cbef7bed3bd737. Report an issue: GitHub.