unslothai/unsloth · warning · SidecarSwapInProgress

A transformers installation is replacing the latest sidecar;

Error message

A transformers installation is replacing the latest sidecar; retry when it completes.

What it means

HTTP 409 from SidecarSwapInProgress: the training start raced against an in-progress transformers sidecar install/swap and lost. The comment in the route explicitly calls it 'a retryable 409, not an internal error' — the fix is to wait for the sidecar installation to complete and retry the same request.

Source

Thrown at studio/backend/routes/training.py:1685

                    backend,
                    reserved_start_request_id,
                    str(exc),
                )
                raise

        try:
            start_task = asyncio.create_task(asyncio.to_thread(_run_backend_start))
            success = await asyncio.shield(start_task)
        except _DiffusionStartInFlight as exc:
            return TrainingJobResponse(
                job_id = "",
                status = "error",
                message = str(exc),
                error = "Diffusion training already active",
            )
        except SidecarSwapInProgress as exc:
            # Expected loss of the race against a sidecar install: a retryable 409, not an internal error.
            raise HTTPException(status_code = 409, detail = str(exc))
        except ExactResumeResourcesUnavailable as exc:
            raise HTTPException(status_code = 409, detail = str(exc))

        if not success:
            progress_error = backend.trainer.training_progress.error
            failure_message = progress_error or "Failed to start training subprocess"
            return TrainingJobResponse(
                job_id = backend.current_job_id or "",
                status = "error",
                message = failure_message,
                error = progress_error or "subprocess_start_failed",
            )

        return TrainingJobResponse(
            job_id = job_id,
            status = "queued",
            message = "Training job queued and starting in subprocess",
            error = None,

View on GitHub (pinned to 203007d190)

Solutions

Wait for the transformers sidecar installation to finish, then retry the identical start request.
Check the installation status endpoint/UI before retrying so you are not racing again.
Serialize runtime swaps and training starts in your automation (never trigger an install between a start attempt and its retry).

Example fix

// before: fire-and-forget start
await post('/training/start', payload)
// after: honor the retryable 409
resp = await post('/training/start', payload)
if resp.status === 409 && /replacing the latest sidecar/.test(resp.detail) {
  await waitForSidecarIdle()
  resp = await post('/training/start', payload)
}

Defensive patterns

Strategy: retry

Validate before calling

const install = await get('/settings/sidecar/status')
if (install.in_progress) await waitForSidecarIdle() // poll until install finishes
await post('/training/start', payload)

Try / catch

async function startWithRetry(payload, attempts = 5) {
  for (;;) {
    try { return await post('/training/start', payload) }
    catch (e) {
      if (e.status === 409 && /replacing the latest sidecar/.test(e.detail) && --attempts > 0) {
        await sleep(10_000); continue
      }
      throw e
    }
  }
}

Prevention

Serialize sidecar installs and training starts — never interleave them in automation.
Check sidecar/installation status before issuing start requests.
Recognize this exact 409 as retryable by message pattern, and back off several seconds between retries.

When it happens

Trigger: POST /training/start (or the resume path) while a 'latest transformers' sidecar installation is concurrently replacing the transformers install; the backend's start sequence detects the swap mid-flight and aborts cleanly.

Common situations: A user (or another tab/automation) triggered a transformers upgrade on the settings page and then immediately started training; CI starting training right after requesting a runtime upgrade.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/bfcc7a4cefcc0722. Report an issue: GitHub.