What causes "Too many training start cancellations are pending" in unslothai/unsloth?

Retry the cancel after a backoff delay (honor 429 semantics; the capacity frees as teardowns complete)

unslothai/unsloth · warning · TrainingStartCancellationCapacityError

Too many training start cancellations are pending

Error message

Too many training start cancellations are pending

What it means

HTTP 429 raised when backend.cancel_start_request throws TrainingStartCancellationCapacityError: the backend has too many pending training-start cancellations and refuses another to bound concurrent teardown work.

Source

Thrown at studio/backend/routes/training.py:1120

    response_model = TrainingStartRequestStatus,
)
async def cancel_training_start_request(
    start_request_id: str = ApiPath(
        ...,
        min_length = 1,
        max_length = 128,
        pattern = TRAINING_REQUEST_ID_PATTERN,
    ),
    current_subject: str = Depends(get_current_subject),
):
    backend = get_training_backend()
    try:
        outcome, record = await asyncio.to_thread(
            backend.cancel_start_request,
            start_request_id,
        )
    except TrainingStartCancellationCapacityError as exc:
        raise HTTPException(status_code = 429, detail = str(exc)) from exc
    if outcome == "superseded":
        raise HTTPException(
            status_code = 409,
            detail = "Training start request no longer owns the current job",
        )
    return _start_request_status_response(record)


def _background_video_generation_active() -> bool:
    """Whether a video clip is generating on the video backend's worker thread.

    POST /video/generate returns at once and generates in the background, so an
    in-flight clip is invisible to the keep-warm in-flight request count the
    API-key training guards consult; ask the backend directly. Best-effort: a
    probe failure must never block a training start."""
    try:
        from core.inference.video import get_video_backend
        return bool(get_video_backend().generate_progress().get("active"))

View on GitHub (pinned to 203007d190)

Solutions

Retry the cancel after a backoff delay (honor 429 semantics; the capacity frees as teardowns complete)
Reduce parallel cancels: serialize cancellation requests in your client
Check the target request's status first - it may already be cancelled and the call unnecessary

Example fix

// before
resp = client.post(f"/training/start-requests/{id}/cancel")  # 429

// after
for attempt in range(5):
    resp = client.post(f"/training/start-requests/{id}/cancel")
    if resp.status_code != 429:
        break
    time.sleep(2 ** attempt)

Defensive patterns

Strategy: retry

Try / catch

resp = client.post(f"/training/start-requests/{rid}/cancel")
if resp.status_code == 429:
    time.sleep(backoff)  # capacity frees as pending teardowns complete
    resp = client.post(f"/training/start-requests/{rid}/cancel")

Prevention

Serialize cancellations instead of cancelling many jobs in parallel
Honor 429 with exponential backoff, never tight-loop retries
Skip cancel when the status already shows cancelled/terminal

When it happens

Trigger: POST /training/start-requests/{id}/cancel while the backend's pending-cancellation slots are all occupied (e.g., several long-running jobs being torn down concurrently).

Common situations: Automation scripts firing cancels for many jobs at once; repeated cancel retries in a tight loop saturating the cancellation queue.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/ec14b2826987b774. Report an issue: GitHub.