{"record":{"id":"ec14b2826987b774","repo":"unslothai/unsloth","slug":"too-many-training-start-cancellations-are-pending-ec14b2","errorCode":null,"errorMessage":"Too many training start cancellations are pending","messagePattern":"Too many training start cancellations are pending","errorType":"http","errorClass":"TrainingStartCancellationCapacityError","httpStatus":429,"severity":"warning","filePath":"studio/backend/routes/training.py","lineNumber":1120,"sourceCode":"    response_model = TrainingStartRequestStatus,\n)\nasync def cancel_training_start_request(\n    start_request_id: str = ApiPath(\n        ...,\n        min_length = 1,\n        max_length = 128,\n        pattern = TRAINING_REQUEST_ID_PATTERN,\n    ),\n    current_subject: str = Depends(get_current_subject),\n):\n    backend = get_training_backend()\n    try:\n        outcome, record = await asyncio.to_thread(\n            backend.cancel_start_request,\n            start_request_id,\n        )\n    except TrainingStartCancellationCapacityError as exc:\n        raise HTTPException(status_code = 429, detail = str(exc)) from exc\n    if outcome == \"superseded\":\n        raise HTTPException(\n            status_code = 409,\n            detail = \"Training start request no longer owns the current job\",\n        )\n    return _start_request_status_response(record)\n\n\ndef _background_video_generation_active() -> bool:\n    \"\"\"Whether a video clip is generating on the video backend's worker thread.\n\n    POST /video/generate returns at once and generates in the background, so an\n    in-flight clip is invisible to the keep-warm in-flight request count the\n    API-key training guards consult; ask the backend directly. Best-effort: a\n    probe failure must never block a training start.\"\"\"\n    try:\n        from core.inference.video import get_video_backend\n        return bool(get_video_backend().generate_progress().get(\"active\"))","sourceCodeStart":1102,"sourceCodeEnd":1138,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/routes/training.py#L1102-L1138","documentation":"HTTP 429 raised when backend.cancel_start_request throws TrainingStartCancellationCapacityError: the backend has too many pending training-start cancellations and refuses another to bound concurrent teardown work.","triggerScenarios":"POST /training/start-requests/{id}/cancel while the backend's pending-cancellation slots are all occupied (e.g., several long-running jobs being torn down concurrently).","commonSituations":"Automation scripts firing cancels for many jobs at once; repeated cancel retries in a tight loop saturating the cancellation queue.","solutions":["Retry the cancel after a backoff delay (honor 429 semantics; the capacity frees as teardowns complete)","Reduce parallel cancels: serialize cancellation requests in your client","Check the target request's status first - it may already be cancelled and the call unnecessary"],"exampleFix":"// before\nresp = client.post(f\"/training/start-requests/{id}/cancel\")  # 429\n\n// after\nfor attempt in range(5):\n    resp = client.post(f\"/training/start-requests/{id}/cancel\")\n    if resp.status_code != 429:\n        break\n    time.sleep(2 ** attempt)","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"resp = client.post(f\"/training/start-requests/{rid}/cancel\")\nif resp.status_code == 429:\n    time.sleep(backoff)  # capacity frees as pending teardowns complete\n    resp = client.post(f\"/training/start-requests/{rid}/cancel\")","preventionTips":["Serialize cancellations instead of cancelling many jobs in parallel","Honor 429 with exponential backoff, never tight-loop retries","Skip cancel when the status already shows cancelled/terminal"],"tags":["training","http-429","rate-limit","cancellation"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}