unslothai/unsloth · error · RuntimeError

job already running

Error message

job already running

What it means

RuntimeError raised under the job manager's lock when starting a new recipe job while self._proc is still alive: the manager is a single-process job runner (one Job, one _proc, one event buffer), so a second concurrent start is refused rather than queued. The check happens after event/job state would otherwise be reset, protecting the running job's state from being clobbered.

Source

Thrown at studio/backend/core/data_recipe/jobs/manager.py:155

        ``internal_api_key_id`` is a workflow-scoped sk-unsloth-* key row id
        minted by the route layer; revoked on terminal state so the key's
        live window is no longer than the run.
        """
        llm_columns = recipe.get("columns") or []
        llm_column_count = 0
        if isinstance(llm_columns, list):
            for column in llm_columns:
                if not isinstance(column, dict):
                    continue
                column_type = str(column.get("column_type") or "").strip().lower()
                if column_type.startswith("llm"):
                    llm_column_count += 1
        if llm_column_count <= 0:
            llm_column_count = 1

        with self._lock:
            if self._proc is not None and self._proc.is_alive():
                raise RuntimeError("job already running")

            job_id = uuid.uuid4().hex
            self._job = Job(job_id = job_id, status = "pending", started_at = time.time())
            self._job.progress_columns_total = llm_column_count
            self._job.source_progress_estimated_total = _github_source_estimated_total(recipe)
            self._job.internal_api_key_id = internal_api_key_id
            self._events.clear()
            self._seq = 0

            run_payload = dict(run)
            run_payload["_job_id"] = job_id
            from utils.native_path_leases import (
                native_path_secret_removed_for_child_start,
                run_without_native_path_secret,
            )
            from utils.hf_cache_settings import child_environment_for_spawn, get_hf_cache_paths

            cache_env = get_hf_cache_paths().child_env({})

View on GitHub (pinned to 203007d190)

Solutions

  1. Query the current job status first and only start when no job is running (or surface 'already running' to the user as a stop-and-restart prompt).
  2. Debounce/disable the Run button in the UI while a job is active.
  3. Make client retries idempotent: do not blindly re-POST the start endpoint on timeout.
  4. If the process is truly stuck, stop/cancel the existing job (or restart the backend) before starting a new one.

Example fix

# before
manager.start(...)  # may raise 'job already running'

# after
status = manager.status()
if status and status.get('running'):
    raise_or_prompt('A job is already running; stop it first')
manager.start(...)
Defensive patterns

Strategy: validation

Validate before calling

job = manager.status()
if job is not None and job.get('status') in ('pending', 'running'):
    raise BusyError('stop the running job before starting another')

Try / catch

try:
    manager.start(...)
except RuntimeError as e:
    if 'job already running' in str(e):
        offer_stop_and_restart(); return

Prevention

When it happens

Trigger: Calling the job start/run endpoint twice in quick succession (double-click, client retry without idempotency); starting a new recipe while a previous long-running recipe job (with LLM columns) is still executing; a zombie child process that is_alive() but stuck.

Common situations: Frontend double-submits; background tab or script retrying on timeout while the first request actually started the job; long LLM-generation jobs where users assume the first attempt failed.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/f22067b07eea6c77. Report an issue: GitHub.