{"record":{"id":"b492fbef5c7ce93e","repo":"unslothai/unsloth","slug":"failed-to-send-command-to-subprocess-exc-b492fb","errorCode":null,"errorMessage":"Failed to send command to subprocess: {exc}","messagePattern":"Failed to send command to subprocess: (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":500,"severity":"error","filePath":"studio/backend/core/inference/orchestrator.py","lineNumber":544,"sourceCode":"                    \" This usually means the system killed it under memory pressure. \"\n                    \"Try a smaller model, lower context length, or close other GPU-heavy apps.\"\n                )\n            return f\"{message}{suffix} Details: pid={pid}, signal={sig_name}, exitcode={exitcode}.\"\n\n        return f\"{message} Details: pid={pid}, exitcode={exitcode}.\"\n\n    # ------------------------------------------------------------------\n    # Queue helpers\n    # ------------------------------------------------------------------\n\n    def _send_cmd(self, cmd: dict) -> None:\n        \"\"\"Send a command to the subprocess.\"\"\"\n        if self._cmd_queue is None:\n            raise RuntimeError(\"No inference subprocess running\")\n        try:\n            self._cmd_queue.put(cmd)\n        except (OSError, ValueError) as exc:\n            raise RuntimeError(f\"Failed to send command to subprocess: {exc}\")\n\n    def _read_resp(self, timeout: float = 1.0) -> Optional[dict]:\n        \"\"\"Read a response from the subprocess (non-blocking with timeout).\"\"\"\n        if self._resp_queue is None:\n            return None\n        try:\n            return self._resp_queue.get(timeout = timeout)\n        except queue.Empty:\n            return None\n        except (EOFError, OSError, ValueError):\n            return None\n\n    def _wait_response(\n        self,\n        expected_type: str,\n        timeout: float = 300.0,\n    ) -> dict:\n        \"\"\"Block until a response of the expected type arrives.","sourceCodeStart":526,"sourceCodeEnd":562,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/inference/orchestrator.py#L526-L562","documentation":"Raised when queue.put(cmd) itself fails with OSError or ValueError: the command queue's underlying pipe is broken (subprocess died and the pipe reader is gone) or the queue was closed. This distinguishes 'queue exists but pipe is dead' from 277's 'no queue at all', both meaning the worker is not reachable.","triggerScenarios":"_send_cmd called while/after the inference subprocess crashed, so the multiprocessing queue's feeder pipe raises BrokenPipeError (an OSError); or ValueError from putting on a closed queue.","commonSituations":"Worker OOM-killed or segfaulted moments before a command was sent; teardown racing a final status poll; long-lived queue whose pipe buffer hit EPIPE after child exit.","solutions":["Treat as worker-down: verify subprocess health, restart it, then resend the command","Add a small retry-with-restart wrapper around command sends that converts this into a supervised restart","Investigate why the subprocess died (see _subprocess_crash_message output elsewhere) to stop recurrence — often OOM"],"exampleFix":"# before\ntry:\n    orchestrator._send_cmd(cmd)\nexcept RuntimeError as e:\n    abort_request()  # user-visible failure\n\n# after\ntry:\n    orchestrator._send_cmd(cmd)\nexcept RuntimeError:\n    await orchestrator.restart_subprocess()\n    orchestrator._send_cmd(cmd)","handlingStrategy":"fallback","validationCode":"if not orchestrator._ensure_subprocess_alive():\n    await orchestrator.restart_subprocess()  # pipe is dead; recreate before send","typeGuard":null,"tryCatchPattern":"try:\n    orchestrator._send_cmd(cmd)\nexcept RuntimeError as exc:\n    if 'Failed to send command' in str(exc):\n        await orchestrator.restart_subprocess()\n        orchestrator._send_cmd(cmd)  # fallback: resend after restart\n    else:\n        raise","preventionTips":["Verify worker liveness immediately before sending, not from a stale cached state","Treat broken queue pipes as worker-down events: restart and resend once","Root-cause worker deaths (OOM, crashes) — broken pipes are the symptom, not the disease"],"tags":["orchestrator","subprocess","ipc","broken-pipe"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}