{"record":{"id":"cbf8b50bea0a6d4b","repo":"unslothai/unsloth","slug":"the-current-inference-worker-did-not-exit-and-stil","errorCode":null,"errorMessage":"The current inference worker did not exit and still holds GPU memory; not starting a new model over it. Retry shortly.","messagePattern":"The current inference worker did not exit and still holds GPU memory; not starting a new model over it\\. Retry shortly\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":500,"severity":"error","filePath":"studio/backend/core/inference/orchestrator.py","lineNumber":1351,"sourceCode":"            )\n\n            if sidecar_swap_kind() == \"repair\":\n                raise SidecarSwapInProgress(\n                    \"A transformers repair is replacing the latest sidecar; \"\n                    \"retry when it completes.\"\n                )\n\n            # Always kill the existing subprocess and spawn fresh: reusing one\n            # after unsloth patches torch internals breaks getsource on reload.\n            if self._ensure_subprocess_alive():\n                self._cancel_generation()\n                time.sleep(0.3)\n                if self._shutdown_subprocess() is False:\n                    # The worker survived terminate/kill (e.g. a wedged CUDA syscall that\n                    # outlives SIGKILL). Its handle is kept, so is_worker_alive() and the\n                    # pre-swap guard still see it; do not spawn a second worker over one\n                    # still holding GPU memory. Fail so the load can retry once it exits.\n                    raise RuntimeError(\n                        \"The current inference worker did not exit and still holds GPU \"\n                        \"memory; not starting a new model over it. Retry shortly.\"\n                    )\n            elif self._proc is not None:\n                self._shutdown_subprocess(timeout = 2)\n\n            disable_xet = sub_config.get(\"disable_xet\", False) or (\n                os.environ.get(\"HF_HUB_DISABLE_XET\") == \"1\"\n            )\n\n            for attempt in range(2):\n                # Stop-loading (/unload -> cancel_load) aborts a load by discarding this\n                # model's loading marker. cancel_load only kills a live child; if the cancel\n                # lands before any child exists (GPU placement, or between retries) there is\n                # nothing to kill, and without this check the loop would spawn a worker and\n                # load the model after /unload reported it unloaded. Observe removal and stop.\n                if model_name not in self.loading_models:\n                    logger.info(","sourceCodeStart":1333,"sourceCodeEnd":1369,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/inference/orchestrator.py#L1333-L1369","documentation":"Raised during the pre-swap teardown in model loading: the existing inference worker survived both SIGTERM and SIGKILL (typically wedged in an unkillable CUDA/driver syscall) and still holds GPU memory. Spawning a new model on top of it would OOM, so the load fails fast and expects a retry once the OS finally reaps the worker.","triggerScenarios":"_shutdown_subprocess() returns False after terminate+kill on a worker stuck in a D-state CUDA syscall; the handle is intentionally kept so liveness checks still see it, and the next load raises this.","commonSituations":"NVIDIA driver bugs, GPU falling off the bus, wedged NCCL/CUDA calls during generation, zombie worker after OOM-killer intervention, or container runtimes delaying SIGKILL on D-state processes.","solutions":["Retry the load shortly — the message says exactly that; the worker usually exits within seconds.","If it never exits, check the process state (ps, D vs Z state) and the GPU with nvidia-smi.","Escalate to a host-level kill (kill -9 the PID from nvidia-smi's process list) if the backend's own kill failed.","If the GPU itself is wedged, a driver/module reload or host reboot is the only fix.","Check dmesg for Xid errors indicating GPU faults."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"import subprocess\ndef gpu_has_stale_worker():\n    out = subprocess.run([\"nvidia-smi\", \"--query-compute-apps=pid\", \"--format=csv,noheader\"], capture_output=True, text=True)\n    return bool(out.stdout.strip())","typeGuard":null,"tryCatchPattern":"try:\n    orchestrator.load_model(name)\nexcept RuntimeError as e:\n    if \"did not exit and still holds GPU memory\" in str(e):\n        time.sleep(10)\n        orchestrator.load_model(name)","preventionTips":["Monitor nvidia-smi for workers stuck after cancels/unloads.","Treat D-state worker processes as a driver-level incident; check dmesg for Xid.","Avoid rapid unload/load cycles that hit wedged CUDA teardown paths."],"tags":["inference","gpu","subprocess","cuda","zombie-process"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}