{"record":{"id":"b049d20b364090b2","repo":"unslothai/unsloth","slug":"llama-server-embedder-failed-to-become-healthy-la","errorCode":null,"errorMessage":"llama-server embedder failed to become healthy. Last output:\n{tail[:2000]}","messagePattern":"llama-server embedder failed to become healthy\\. Last output:\n(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"studio/backend/core/rag/embed_llama_server.py","lineNumber":615,"sourceCode":"            **windows_hidden_subprocess_kwargs(),\n            **child_popen_kwargs(),\n        )\n        self._process = proc\n        # Long-lived, and child_popen_kwargs() is empty on macOS, so the crash\n        # record is the only thing that can reap it after a force quit.\n        adopt_pid(proc.pid)\n        self._port = port\n        self._stdout_thread = threading.Thread(\n            target = self._drain_stdout,\n            args = (proc,),\n            daemon = True,\n            name = \"llama-embed-stdout\",\n        )\n        self._stdout_thread.start()\n        if not self._wait_for_health(config.EMBED_STARTUP_TIMEOUT_S):\n            tail = \"\\n\".join(self._stdout_lines[-30:])\n            self._kill_process()\n            raise RuntimeError(\n                f\"llama-server embedder failed to become healthy. Last output:\\n{tail[:2000]}\"\n            )\n\n    @staticmethod\n    def _find_free_port() -> int:\n        from core.inference.llama_cpp import LlamaCppBackend\n        return LlamaCppBackend._find_free_port()\n\n    def _wait_for_health(\n        self,\n        timeout: float,\n        interval: float = 0.5,\n    ) -> bool:\n        \"\"\"Poll /health until 200; bail early if the process exits.\"\"\"\n        deadline = time.monotonic() + timeout\n        url = f\"{self._base_url}/health\"\n        while time.monotonic() < deadline:\n            if not self._process_alive():","sourceCodeStart":597,"sourceCodeEnd":633,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/rag/embed_llama_server.py#L597-L633","documentation":"Raised when the llama-server embedding subprocess fails its health check within EMBED_STARTUP_TIMEOUT_S after being spawned. The backend starts a llama-server process, drains its stdout in a daemon thread, polls /health, and if the server never becomes ready it kills the process and re-raises with the last 30 stdout lines (capped at 2000 chars) as diagnostics. Typical root causes are a corrupt/unsupported GGUF file, a missing binary, or startup flags the installed llama-server rejects.","triggerScenarios":"Constructing LlamaServerBackend (or first encode/dim call that triggers _ensure_ready -> spawn) with config.EMBED_MODEL_PATH pointing at a bad GGUF, an incompatible llama-server binary, insufficient VRAM for the configured n_gpu_layers, or EMBED_STARTUP_TIMEOUT_S set too low for a large model to load.","commonSituations":"Switching RAG_EMBED_BACKEND to 'llama-server' with a GGUF that is not an embedding model (e.g. a chat model without an embedding head); upgrading llama.cpp which changed CLI flags; first-cold-start on a slow disk where model loading exceeds the timeout; CUDA driver mismatch causing the subprocess to abort at startup.","solutions":["Read the 'Last output' tail in the message — it contains llama-server's own startup error (e.g. 'unknown argument', 'GGUF header mismatch', CUDA errors).","Verify the GGUF is an embedding model and the path in EMBED_MODEL_PATH exists: run llama-server manually with the same flags the backend uses.","Increase EMBED_STARTUP_TIMEOUT_S for large models on slow storage.","If torch/sentence-transformers works on this machine, set RAG_EMBED_BACKEND=sentence-transformers to bypass llama-server entirely.","Reinstall/upgrade the bundled llama-server binary so its CLI matches what the backend expects."],"exampleFix":"# before\nRAG_EMBED_STARTUP_TIMEOUT_S=15  # too short for a 7B GGUF on cold disk\n\n# after\nRAG_EMBED_STARTUP_TIMEOUT_S=120  # large embedding GGUFs can take minutes to mmap+warm","handlingStrategy":"fallback","validationCode":"import shutil\nfrom core import config\n\ndef llama_server_precheck() -> list[str]:\n    problems = []\n    model = getattr(config, \"EMBED_MODEL_PATH\", None)\n    if not model or not Path(model).is_file():\n        problems.append(f\"EMBED_MODEL_PATH missing or not a file: {model!r}\")\n    if shutil.which(\"llama-server\") is None and not getattr(config, \"LLAMA_SERVER_BIN\", \"\"):\n        problems.append(\"llama-server binary not found on PATH\")\n    return problems  # empty list -> safe to construct LlamaServerBackend","typeGuard":null,"tryCatchPattern":"try:\n    backend = LlamaServerBackend()\nexcept RuntimeError as e:\n    if \"failed to become healthy\" not in str(e):\n        raise\n    log.error(\"embedder startup failed; server output: %s\", e)\n    backend = build_sentence_transformers_backend()  # fallback","preventionTips":["Smoke-test the llama-server command line manually whenever EMBED_MODEL_PATH or the binary changes.","Size EMBED_STARTUP_TIMEOUT_S to the model load time on your slowest storage, not your dev box.","Keep a known-good embedding GGUF pinned by checksum so a bad download cannot reach production."],"tags":["subprocess","llama-cpp","embeddings","startup","timeout"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}