{"record":{"id":"d7cbef7bed3bd737","repo":"unslothai/unsloth","slug":"training-state-changed-during-status-read","errorCode":null,"errorMessage":"Training state changed during status read","messagePattern":"Training state changed during status read","errorType":"http","errorClass":"HTTPException","httpStatus":409,"severity":"info","filePath":"studio/backend/routes/training.py","lineNumber":1993,"sourceCode":"\n\n@router.get(\"/status\")\nasync def get_training_status(current_subject: str = Depends(get_current_subject)):\n    \"\"\"\n    Get the current training status.\n    \"\"\"\n    try:\n        backend = get_training_backend()\n        for _ in range(3):\n            identity_before = _training_status_identity(backend)\n            is_active = await asyncio.to_thread(_run_active, backend)\n            identity = _training_status_identity(backend)\n            if identity != identity_before:\n                continue\n            status = _build_training_status(backend, identity, is_active)\n            if _training_status_identity(backend) == identity:\n                return status\n        raise HTTPException(status_code = 409, detail = \"Training state changed during status read\")\n    except HTTPException:\n        raise\n    except Exception as e:\n        raise log_and_http_error(\n            e,\n            500,\n            \"Failed to get training status\",\n            event = \"training.status_failed\",\n            log = logger,\n        )\n\n\n@router.get(\"/metrics\", response_model = TrainingMetricsResponse)\nasync def get_training_metrics(\n    expected_job_id: Optional[str] = None, current_subject: str = Depends(get_current_subject)\n):\n    \"\"\"\n    Get training metrics (loss, learning rate, steps).","sourceCodeStart":1975,"sourceCodeEnd":2011,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/routes/training.py#L1975-L2011","documentation":"HTTP 409 from the status endpoint: it builds a consistent training-status snapshot by comparing a backend identity token before/after each read, retrying up to 3 times. If the identity (job id / spawn state) changed during every attempt — i.e. a training job started, stopped, or was replaced concurrently with each read — it gives up and returns 409 rather than a torn snapshot.","triggerScenarios":"GET /training/status issued at the exact moment a training job starts, stops, resets, or is superseded, such that all 3 read attempts straddle a transition.","commonSituations":"Aggressive status polling (sub-second intervals) while a job is being started/stopped from another tab or by automation; slow reads on large histories making the race window wider.","solutions":["Retry the GET after a short delay — the next read usually observes a stable state.","Space out polling (>= 1-2s) and stop polling during explicit start/stop transitions.","Key subsequent requests off the job_id in the successful status response to detect supersession cleanly."],"exampleFix":"// before\nconst status = await get('/training/status')\n// after\nlet status\nfor (let i = 0; i < 5; i++) {\n  try { status = await get('/training/status'); break }\n  catch (e) { if (e.status !== 409) throw e; await sleep(1000) }\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"async function getStatusSafe() {\n  for (let i = 0; i < 5; i++) {\n    try { return await get('/training/status') }\n    catch (e) { if (e.status === 409 && /state changed/.test(e.detail)) { await sleep(1000); continue } throw e }\n  }\n  throw new Error('Training status unstable after 5 attempts')\n}","preventionTips":["Poll status at 1-2s intervals, not sub-second.","Pause polling while a start/stop/reset request is in flight.","Never build UI state from a torn snapshot — always use the returned job_id as your consistency key."],"tags":["status","race-condition","snapshot","retryable","http-409"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}