unslothai/unsloth · error · RuntimeError

Failed to send command to subprocess: {exc}

Error message

Failed to send command to subprocess: {exc}

What it means

Raised by ExportOrchestrator._send_cmd when queue.put() raises OSError or ValueError. ValueError occurs when the multiprocessing Queue has been closed (e.g. after the worker died and the queue's underlying pipe/flush was torn down); OSError indicates a broken pipe to the dead worker. Practically it means the worker died between the liveness check and the put, or the queue was closed.

Source

Thrown at studio/backend/core/export/orchestrator.py:334

        """atexit handler."""
        self._shutdown_subprocess(timeout = 5.0)

    def _ensure_subprocess_alive(self) -> bool:
        """Check if subprocess is alive."""
        return self._proc is not None and self._proc.is_alive()

    # ------------------------------------------------------------------
    # Queue helpers
    # ------------------------------------------------------------------

    def _send_cmd(self, cmd: dict) -> None:
        """Send a command to the subprocess."""
        if self._cmd_queue is None:
            raise RuntimeError("No export subprocess running")
        try:
            self._cmd_queue.put(cmd)
        except (OSError, ValueError) as exc:
            raise RuntimeError(f"Failed to send command to subprocess: {exc}")

    def _read_resp(self, timeout: float = 1.0) -> Optional[dict]:
        """Read a response from the subprocess (non-blocking with timeout)."""
        if self._resp_queue is None:
            return None
        try:
            return self._resp_queue.get(timeout = timeout)
        except queue.Empty:
            return None
        except (EOFError, OSError, ValueError):
            return None

    def _wait_response(
        self,
        expected_type: str,
        timeout: float = 3600.0,
    ) -> dict:
        """Block until a response of the expected type arrives.

View on GitHub (pinned to 203007d190)

Solutions

1. Check the worker's exit status and the backend log (worker stdout/stderr is forwarded as 'log' events) to find why the subprocess died — OOM and CUDA errors are the usual causes
2. Catch this RuntimeError at the op level, call _shutdown_subprocess(), then retry the whole op (checkpoint load + export) once
3. If OOM: reduce merge/quant sizes, free GPU memory before export, or lower parallelism
4. Serialize export start/stop with a lock so shutdown cannot interleave with _send_cmd

Example fix

# before
self._send_cmd(cmd)

# after
try:
    self._send_cmd(cmd)
except RuntimeError:
    self._shutdown_subprocess()
    ok, msg = self.load_checkpoint(self._last_load_params)
    if not ok:
        raise
    self._send_cmd(cmd)

try:
    orch._send_cmd(cmd)
except RuntimeError as e:
    if "Failed to send command" in str(e):
        orch._shutdown_subprocess()
        ok, msg = orch.load_checkpoint(last_load_params)
        if ok:
            orch._send_cmd(cmd)  # one retry after respawn
        else:
            raise

When it happens

Trigger: Worker process crashes (OOM-killed, segfault in GGUF conversion, CUDA abort) while a command is being put; _shutdown_subprocess() closing _cmd_queue concurrently with _send_cmd(); queue buffer/pipe invalidated after a prior worker crash that was not fully cleaned up.

Common situations: Export worker killed by the Linux OOM killer during large-model GGUF merge; GPU driver reset killing the worker mid-put; concurrent export + shutdown (user cancels while a command is in flight); stale orchestrator state after a previous crash.

Related errors

• Export subprocess crashed during wait
• No export subprocess running
• Failed to send command to subprocess: {exc}
• The inference worker stopped unexpectedly while loading the
• Failed to export GGUF model

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/452432dc2eea8f84. Report an issue: GitHub.