unslothai/unsloth · error · RuntimeError

The current inference worker did not exit and still holds GP

Error message

The current inference worker did not exit and still holds GPU memory; not starting a new model over it. Retry shortly.

What it means

Raised during the pre-swap teardown in model loading: the existing inference worker survived both SIGTERM and SIGKILL (typically wedged in an unkillable CUDA/driver syscall) and still holds GPU memory. Spawning a new model on top of it would OOM, so the load fails fast and expects a retry once the OS finally reaps the worker.

Source

Thrown at studio/backend/core/inference/orchestrator.py:1351

            )

            if sidecar_swap_kind() == "repair":
                raise SidecarSwapInProgress(
                    "A transformers repair is replacing the latest sidecar; "
                    "retry when it completes."
                )

            # Always kill the existing subprocess and spawn fresh: reusing one
            # after unsloth patches torch internals breaks getsource on reload.
            if self._ensure_subprocess_alive():
                self._cancel_generation()
                time.sleep(0.3)
                if self._shutdown_subprocess() is False:
                    # The worker survived terminate/kill (e.g. a wedged CUDA syscall that
                    # outlives SIGKILL). Its handle is kept, so is_worker_alive() and the
                    # pre-swap guard still see it; do not spawn a second worker over one
                    # still holding GPU memory. Fail so the load can retry once it exits.
                    raise RuntimeError(
                        "The current inference worker did not exit and still holds GPU "
                        "memory; not starting a new model over it. Retry shortly."
                    )
            elif self._proc is not None:
                self._shutdown_subprocess(timeout = 2)

            disable_xet = sub_config.get("disable_xet", False) or (
                os.environ.get("HF_HUB_DISABLE_XET") == "1"
            )

            for attempt in range(2):
                # Stop-loading (/unload -> cancel_load) aborts a load by discarding this
                # model's loading marker. cancel_load only kills a live child; if the cancel
                # lands before any child exists (GPU placement, or between retries) there is
                # nothing to kill, and without this check the loop would spawn a worker and
                # load the model after /unload reported it unloaded. Observe removal and stop.
                if model_name not in self.loading_models:
                    logger.info(

View on GitHub (pinned to 203007d190)

Solutions

Retry the load shortly — the message says exactly that; the worker usually exits within seconds.
If it never exits, check the process state (ps, D vs Z state) and the GPU with nvidia-smi.
Escalate to a host-level kill (kill -9 the PID from nvidia-smi's process list) if the backend's own kill failed.
If the GPU itself is wedged, a driver/module reload or host reboot is the only fix.
Check dmesg for Xid errors indicating GPU faults.

Defensive patterns

Strategy: retry

Validate before calling

import subprocess
def gpu_has_stale_worker():
    out = subprocess.run(["nvidia-smi", "--query-compute-apps=pid", "--format=csv,noheader"], capture_output=True, text=True)
    return bool(out.stdout.strip())

Try / catch

try:
    orchestrator.load_model(name)
except RuntimeError as e:
    if "did not exit and still holds GPU memory" in str(e):
        time.sleep(10)
        orchestrator.load_model(name)

Prevention

Monitor nvidia-smi for workers stuck after cancels/unloads.
Treat D-state worker processes as a driver-level incident; check dmesg for Xid.
Avoid rapid unload/load cycles that hit wedged CUDA teardown paths.

When it happens

Trigger: _shutdown_subprocess() returns False after terminate+kill on a worker stuck in a D-state CUDA syscall; the handle is intentionally kept so liveness checks still see it, and the next load raises this.

Common situations: NVIDIA driver bugs, GPU falling off the bus, wedged NCCL/CUDA calls during generation, zombie worker after OOM-killer intervention, or container runtimes delaying SIGKILL on D-state processes.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/cbf8b50bea0a6d4b. Report an issue: GitHub.