{"record":{"id":"ed40bfbd8f55e4fc","repo":"unslothai/unsloth","slug":"timeout-waiting-for-expected-type-response-aft","errorCode":null,"errorMessage":"Timeout waiting for '{expected_type}' response after {timeout}s","messagePattern":"Timeout waiting for '(.+?)' response after (.+?)s","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"studio/backend/core/export/orchestrator.py","lineNumber":404,"sourceCode":"                if message:\n                    logger.info(\"export_progress\", phase = message)\n                    self._append_log(\n                        {\n                            \"stream\": \"status\",\n                            \"line\": message,\n                            \"ts\": resp.get(\"ts\", time.time()),\n                        }\n                    )\n                continue\n\n            # Other response types during wait — skip.\n            logger.debug(\n                \"Skipping response type '%s' while waiting for '%s'\",\n                rtype,\n                expected_type,\n            )\n\n        raise RuntimeError(f\"Timeout waiting for '{expected_type}' response after {timeout}s\")\n\n    def _drain_queue(self) -> list:\n        \"\"\"Drain all pending responses.\"\"\"\n        events = []\n        if self._resp_queue is None:\n            return events\n        while True:\n            try:\n                events.append(self._resp_queue.get_nowait())\n            except queue.Empty:\n                return events\n            except (EOFError, OSError, ValueError):\n                return events\n\n    # ------------------------------------------------------------------\n    # Public API — same interface as ExportBackend\n    # ------------------------------------------------------------------\n","sourceCodeStart":386,"sourceCodeEnd":422,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/export/orchestrator.py#L386-L422","documentation":"Raised by ExportOrchestrator._wait_response when the deadline (timeout, default scaled to 3600s × number of quants) expires without receiving the expected response type (e.g. 'export_gguf_done'). Unlike a crash, the worker is still alive but silent — it is either genuinely still working or wedged (e.g. stuck in a CUDA syscall or I/O).","triggerScenarios":"GGUF conversion of a 30B+ model with a multi-quant list exceeding 3600s×n; worker wedged on extremely slow disk I/O writing multi-GB outputs; worker deadlocked on GPU contention with another process; response lost because an earlier handler consumed it.","commonSituations":"Very large models on slow spinning disks or network storage; multiple heavy GPU jobs sharing one device; host under memory pressure causing swap-thrash during merge; a first export on a cold page cache.","solutions":["Check whether the worker is actually progressing: the orchestrator forwards 'status'/'log' events as export_progress lines in the server log — if they advance, just increase the timeout/retry","Free GPU/CPU/disk contention (stop training or other inference) and retry the export","Split a multi-quant export into individual runs so each gets the full hour","If no progress lines at all appear, the worker is wedged — kill it (shutdown_subprocess) and retry once from a fresh checkpoint load","Ensure adequate free disk space; a full disk can stall writes near the timeout"],"exampleFix":"# before\nresp = self._wait_response(f\"export_{export_type}_done\", timeout=3600)\n\n# after\nresp = self._wait_response(\n    f\"export_{export_type}_done\",\n    timeout=3600 * max(1, n_quants),\n)","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    resp = orch._wait_response(expected, timeout=timeout)\nexcept RuntimeError as e:\n    if \"Timeout waiting for\" in str(e):\n        if worker_made_progress_recently():\n            resp = orch._wait_response(expected, timeout=timeout)  # extend once\n        else:\n            orch._shutdown_subprocess()  # wedged; recycle and restart op\n            raise","preventionTips":["Scale timeouts by quant count and model size as the code does","Run multi-quant exports as separate single-quant ops when near the hour limit","Keep the export device free of competing GPU work"],"tags":["export","timeout","gguf","large-models"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}