{"record":{"id":"452432dc2eea8f84","repo":"unslothai/unsloth","slug":"failed-to-send-command-to-subprocess-exc","errorCode":null,"errorMessage":"Failed to send command to subprocess: {exc}","messagePattern":"Failed to send command to subprocess: (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"studio/backend/core/export/orchestrator.py","lineNumber":334,"sourceCode":"        \"\"\"atexit handler.\"\"\"\n        self._shutdown_subprocess(timeout = 5.0)\n\n    def _ensure_subprocess_alive(self) -> bool:\n        \"\"\"Check if subprocess is alive.\"\"\"\n        return self._proc is not None and self._proc.is_alive()\n\n    # ------------------------------------------------------------------\n    # Queue helpers\n    # ------------------------------------------------------------------\n\n    def _send_cmd(self, cmd: dict) -> None:\n        \"\"\"Send a command to the subprocess.\"\"\"\n        if self._cmd_queue is None:\n            raise RuntimeError(\"No export subprocess running\")\n        try:\n            self._cmd_queue.put(cmd)\n        except (OSError, ValueError) as exc:\n            raise RuntimeError(f\"Failed to send command to subprocess: {exc}\")\n\n    def _read_resp(self, timeout: float = 1.0) -> Optional[dict]:\n        \"\"\"Read a response from the subprocess (non-blocking with timeout).\"\"\"\n        if self._resp_queue is None:\n            return None\n        try:\n            return self._resp_queue.get(timeout = timeout)\n        except queue.Empty:\n            return None\n        except (EOFError, OSError, ValueError):\n            return None\n\n    def _wait_response(\n        self,\n        expected_type: str,\n        timeout: float = 3600.0,\n    ) -> dict:\n        \"\"\"Block until a response of the expected type arrives.","sourceCodeStart":316,"sourceCodeEnd":352,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/export/orchestrator.py#L316-L352","documentation":"Raised by ExportOrchestrator._send_cmd when queue.put() raises OSError or ValueError. ValueError occurs when the multiprocessing Queue has been closed (e.g. after the worker died and the queue's underlying pipe/flush was torn down); OSError indicates a broken pipe to the dead worker. Practically it means the worker died between the liveness check and the put, or the queue was closed.","triggerScenarios":"Worker process crashes (OOM-killed, segfault in GGUF conversion, CUDA abort) while a command is being put; _shutdown_subprocess() closing _cmd_queue concurrently with _send_cmd(); queue buffer/pipe invalidated after a prior worker crash that was not fully cleaned up.","commonSituations":"Export worker killed by the Linux OOM killer during large-model GGUF merge; GPU driver reset killing the worker mid-put; concurrent export + shutdown (user cancels while a command is in flight); stale orchestrator state after a previous crash.","solutions":["Check the worker's exit status and the backend log (worker stdout/stderr is forwarded as 'log' events) to find why the subprocess died — OOM and CUDA errors are the usual causes","Catch this RuntimeError at the op level, call _shutdown_subprocess(), then retry the whole op (checkpoint load + export) once","If OOM: reduce merge/quant sizes, free GPU memory before export, or lower parallelism","Serialize export start/stop with a lock so shutdown cannot interleave with _send_cmd"],"exampleFix":"# before\nself._send_cmd(cmd)\n\n# after\ntry:\n    self._send_cmd(cmd)\nexcept RuntimeError:\n    self._shutdown_subprocess()\n    ok, msg = self.load_checkpoint(self._last_load_params)\n    if not ok:\n        raise\n    self._send_cmd(cmd)","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    orch._send_cmd(cmd)\nexcept RuntimeError as e:\n    if \"Failed to send command\" in str(e):\n        orch._shutdown_subprocess()\n        ok, msg = orch.load_checkpoint(last_load_params)\n        if ok:\n            orch._send_cmd(cmd)  # one retry after respawn\n        else:\n            raise","preventionTips":["Monitor worker memory; kill exports approaching host RAM limits before the OOM killer does","Free GPU memory before starting exports","Wrap every export op in a supervisor that respawns the worker once on death"],"tags":["export","subprocess","queue","broken-pipe","oom"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}