unslothai/unsloth · info · HTTPException

Training job was superseded

Error message

Training job was superseded

What it means

HTTP 409 from GET /training/metrics — first check: the metrics request is invalid because a new job spawn is in progress (backend._new_job_spawn_id set) or the optional expected_job_id does not match the backend's current_job_id. This guards clients against silently mixing metrics from two different runs.

Source

Thrown at studio/backend/routes/training.py:2019

            event = "training.status_failed",
            log = logger,
        )


@router.get("/metrics", response_model = TrainingMetricsResponse)
async def get_training_metrics(
    expected_job_id: Optional[str] = None, current_subject: str = Depends(get_current_subject)
):
    """
    Get training metrics (loss, learning rate, steps).
    """
    try:
        backend = get_training_backend()
        job_id = getattr(backend, "current_job_id", "") or ""
        if getattr(backend, "_new_job_spawn_id", None) is not None or (
            expected_job_id is not None and expected_job_id != job_id
        ):
            raise HTTPException(status_code = 409, detail = "Training job was superseded")

        loss_history = list(backend.loss_history)
        lr_history = list(backend.lr_history)
        step_history = list(backend.step_history)
        grad_norm_history = list(getattr(backend, "grad_norm_history", []))
        grad_norm_step_history = list(getattr(backend, "grad_norm_step_history", []))

        if (
            getattr(backend, "_new_job_spawn_id", None) is not None
            or (getattr(backend, "current_job_id", "") or "") != job_id
        ):
            raise HTTPException(status_code = 409, detail = "Training job was superseded")

        current_loss = loss_history[-1] if loss_history else None
        current_lr = lr_history[-1] if lr_history else None
        current_step = step_history[-1] if step_history else None

        return TrainingMetricsResponse(

View on GitHub (pinned to 203007d190)

Solutions

1. Refetch /training/status, take its job_id, and re-request metrics with that id.
2. If a spawn is in flight, wait for the start request to complete before polling metrics.
3. Treat this 409 as a signal to reset local metric history — the new run's series starts fresh.

const status = await get('/training/status')
if (!status.job_id || status.spawning) throw new Error('No stable job to read metrics for')
const metrics = await get(`/training/metrics?expected_job_id=${status.job_id}`)

try { await get('/training/metrics', {params: {expected_job_id: jobId}}) } catch (e) { if (e.status === 409 && /superseded/.test(e.detail)) { const s = await get('/training/status'); resetLocalCharts(s.job_id) } else throw e }

When it happens

Trigger: GET /training/metrics?expected_job_id=X where X differs from the running job's id, or any metrics fetch while a new training job is mid-spawn.

Common situations: A dashboard holding a job id from a previous run after a restart/resume spawned a new one; polling metrics immediately after issuing a start request.

Related errors

• The requested training job is no longer active.
• Training start request no longer owns the current job
• A transformers installation is in progress. Retry when it co
• A transformers installation is replacing the latest sidecar;
• Knowledge base is being deleted

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/f2b4c825e547d2d0. Report an issue: GitHub.