{"record":{"id":"ac99ad89d9766159","repo":"unslothai/unsloth","slug":"export-subprocess-crashed-during-wait","errorCode":null,"errorMessage":"Export subprocess crashed during wait","messagePattern":"Export subprocess crashed during wait","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"studio/backend/core/export/orchestrator.py","lineNumber":365,"sourceCode":"    def _wait_response(\n        self,\n        expected_type: str,\n        timeout: float = 3600.0,\n    ) -> dict:\n        \"\"\"Block until a response of the expected type arrives.\n\n        Export ops can take a long time — GGUF conversion for large\n        models (30B+) easily takes 20-30 minutes. Default timeout 1 hour.\n        \"\"\"\n        deadline = time.monotonic() + timeout\n\n        while time.monotonic() < deadline:\n            remaining = max(0.1, deadline - time.monotonic())\n            resp = self._read_resp(timeout = min(remaining, 2.0))\n\n            if resp is None:\n                if not self._ensure_subprocess_alive():\n                    raise RuntimeError(\"Export subprocess crashed during wait\")\n                continue\n\n            rtype = resp.get(\"type\", \"\")\n\n            if rtype == expected_type:\n                return resp\n\n            if rtype == \"error\":\n                error_msg = resp.get(\"error\", \"Unknown error\")\n                raise RuntimeError(f\"Subprocess error: {error_msg}\")\n\n            if rtype == \"log\":\n                # Forwarded stdout/stderr line from the worker.\n                self._append_log(resp)\n                continue\n\n            if rtype == \"status\":\n                message = resp.get(\"message\", \"\")","sourceCodeStart":347,"sourceCodeEnd":383,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/export/orchestrator.py#L347-L383","documentation":"Raised by ExportOrchestrator._wait_response when a read from the response queue times out (2s poll) AND _ensure_subprocess_alive() reports the worker process is gone. The export did not fail gracefully — the worker process itself terminated (crash, OOM kill, SIGKILL) while the orchestrator was waiting for its done/error response.","triggerScenarios":"Worker OOM-killed during GGUF conversion of a 30B+ model; segfault or CUDA fatal error inside the worker during merge/quantize; worker calling os._exit or dying from a wedged CUDA syscall; SIGKILL from an external supervisor.","commonSituations":"Host RAM exhausted by the merge step (merge writes multi-GB safetensors); GPU memory leak across repeated exports finally killing the worker; driver/driver-state issues after hibernate or another process crashing the GPU context.","solutions":["Look at dmesg/journalctl for an OOM-kill entry for the export worker; if present, free memory (close other models, lower merge precision) or add swap/RAM before retrying","Check forwarded worker logs ('log' events in the server log) for a Python traceback or CUDA error immediately before the crash","Retry the export once after a fresh checkpoint load — the orchestrator kills and respawns the worker, clearing wedged GPU state","If a survivor holds GPU memory (the shutdown path warns about this), wait for it to exit before retrying"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    resp = orch._wait_response(\"export_gguf_done\", timeout=3600)\nexcept RuntimeError as e:\n    if \"crashed during wait\" in str(e):\n        # inspect dmesg for OOM, then full retry: load + export\n        raise","preventionTips":["Keep enough free RAM/disk for the merge step of large-model GGUF exports","Watch export_progress log lines; absence of progress for minutes signals a wedged worker","Run one heavy GPU job at a time on the export device"],"tags":["export","subprocess","crash","oom","gpu"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}