unslothai/unsloth · error · RuntimeError

Timeout waiting for '{expected_type}' response after {timeou

Error message

Timeout waiting for '{expected_type}' response after {timeout}s

What it means

Raised by ExportOrchestrator._wait_response when the deadline (timeout, default scaled to 3600s × number of quants) expires without receiving the expected response type (e.g. 'export_gguf_done'). Unlike a crash, the worker is still alive but silent — it is either genuinely still working or wedged (e.g. stuck in a CUDA syscall or I/O).

Source

Thrown at studio/backend/core/export/orchestrator.py:404

                if message:
                    logger.info("export_progress", phase = message)
                    self._append_log(
                        {
                            "stream": "status",
                            "line": message,
                            "ts": resp.get("ts", time.time()),
                        }
                    )
                continue

            # Other response types during wait — skip.
            logger.debug(
                "Skipping response type '%s' while waiting for '%s'",
                rtype,
                expected_type,
            )

        raise RuntimeError(f"Timeout waiting for '{expected_type}' response after {timeout}s")

    def _drain_queue(self) -> list:
        """Drain all pending responses."""
        events = []
        if self._resp_queue is None:
            return events
        while True:
            try:
                events.append(self._resp_queue.get_nowait())
            except queue.Empty:
                return events
            except (EOFError, OSError, ValueError):
                return events

    # ------------------------------------------------------------------
    # Public API — same interface as ExportBackend
    # ------------------------------------------------------------------

View on GitHub (pinned to 203007d190)

Solutions

1. Check whether the worker is actually progressing: the orchestrator forwards 'status'/'log' events as export_progress lines in the server log — if they advance, just increase the timeout/retry
2. Free GPU/CPU/disk contention (stop training or other inference) and retry the export
3. Split a multi-quant export into individual runs so each gets the full hour
4. If no progress lines at all appear, the worker is wedged — kill it (shutdown_subprocess) and retry once from a fresh checkpoint load
5. Ensure adequate free disk space; a full disk can stall writes near the timeout

Example fix

# before
resp = self._wait_response(f"export_{export_type}_done", timeout=3600)

# after
resp = self._wait_response(
    f"export_{export_type}_done",
    timeout=3600 * max(1, n_quants),
)

try:
    resp = orch._wait_response(expected, timeout=timeout)
except RuntimeError as e:
    if "Timeout waiting for" in str(e):
        if worker_made_progress_recently():
            resp = orch._wait_response(expected, timeout=timeout)  # extend once
        else:
            orch._shutdown_subprocess()  # wedged; recycle and restart op
            raise

When it happens

Trigger: GGUF conversion of a 30B+ model with a multi-quant list exceeding 3600s×n; worker wedged on extremely slow disk I/O writing multi-GB outputs; worker deadlocked on GPU contention with another process; response lost because an earlier handler consumed it.

Common situations: Very large models on slow spinning disks or network storage; multiple heavy GPU jobs sharing one device; host under memory pressure causing swap-thrash during merge; a first export on a cold page cache.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• GGUF conversion produced a symlink, refusing to relocate it:
• GGUF conversion produced no files: no .gguf outputs for {abs
• GGUF conversion produced a symlinked Modelfile, refusing to
• Subprocess error: {error_msg}
• Failed to export GGUF model

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/ed40bfbd8f55e4fc. Report an issue: GitHub.