unslothai/unsloth · error · RuntimeError

Export subprocess crashed during wait

Error message

Export subprocess crashed during wait

What it means

Raised by ExportOrchestrator._wait_response when a read from the response queue times out (2s poll) AND _ensure_subprocess_alive() reports the worker process is gone. The export did not fail gracefully — the worker process itself terminated (crash, OOM kill, SIGKILL) while the orchestrator was waiting for its done/error response.

Source

Thrown at studio/backend/core/export/orchestrator.py:365

    def _wait_response(
        self,
        expected_type: str,
        timeout: float = 3600.0,
    ) -> dict:
        """Block until a response of the expected type arrives.

        Export ops can take a long time — GGUF conversion for large
        models (30B+) easily takes 20-30 minutes. Default timeout 1 hour.
        """
        deadline = time.monotonic() + timeout

        while time.monotonic() < deadline:
            remaining = max(0.1, deadline - time.monotonic())
            resp = self._read_resp(timeout = min(remaining, 2.0))

            if resp is None:
                if not self._ensure_subprocess_alive():
                    raise RuntimeError("Export subprocess crashed during wait")
                continue

            rtype = resp.get("type", "")

            if rtype == expected_type:
                return resp

            if rtype == "error":
                error_msg = resp.get("error", "Unknown error")
                raise RuntimeError(f"Subprocess error: {error_msg}")

            if rtype == "log":
                # Forwarded stdout/stderr line from the worker.
                self._append_log(resp)
                continue

            if rtype == "status":
                message = resp.get("message", "")

View on GitHub (pinned to 203007d190)

Solutions

1. Look at dmesg/journalctl for an OOM-kill entry for the export worker; if present, free memory (close other models, lower merge precision) or add swap/RAM before retrying
2. Check forwarded worker logs ('log' events in the server log) for a Python traceback or CUDA error immediately before the crash
3. Retry the export once after a fresh checkpoint load — the orchestrator kills and respawns the worker, clearing wedged GPU state
4. If a survivor holds GPU memory (the shutdown path warns about this), wait for it to exit before retrying

try:
    resp = orch._wait_response("export_gguf_done", timeout=3600)
except RuntimeError as e:
    if "crashed during wait" in str(e):
        # inspect dmesg for OOM, then full retry: load + export
        raise

When it happens

Trigger: Worker OOM-killed during GGUF conversion of a 30B+ model; segfault or CUDA fatal error inside the worker during merge/quantize; worker calling os._exit or dying from a wedged CUDA syscall; SIGKILL from an external supervisor.

Common situations: Host RAM exhausted by the merge step (merge writes multi-GB safetensors); GPU memory leak across repeated exports finally killing the worker; driver/driver-state issues after hibernate or another process crashing the GPU context.

Related errors

• Failed to send command to subprocess: {exc}
• The inference worker stopped unexpectedly while loading the
• No export subprocess running
• The current inference worker did not exit and still holds GP
• Failed to export GGUF model

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/ac99ad89d9766159. Report an issue: GitHub.