unslothai/unsloth · error · RuntimeError

llama-server embedder POST {path} failed after retry

Error message

llama-server embedder POST {path} failed after retry

What it means

Raised when a POST to the llama-server embedder fails with a transport-level error (connection refused/reset, timeout) on both attempts — the first failure triggers a full _restart() of the subprocess, and the second failure exhausts the retry loop. The original transport exception is chained as __cause__. It means the llama-server process is dying or unreachable even after a clean respawn.

Source

Thrown at studio/backend/core/rag/embed_llama_server.py:728

        wedges a request); a fresh server unsticks both."""
        last_exc: Exception | None = None
        for attempt in range(2):
            self._ensure_ready()
            try:
                resp = self._client.post(f"{self._base_url}{path}", json = payload)
                resp.raise_for_status()
                return resp.json()
            except (*_TRANSPORT_ERRORS, httpx.TimeoutException) as e:
                last_exc = e
                if attempt == 0:
                    self._restart()
                    continue
            except httpx.HTTPStatusError as e:
                body = e.response.text[:500] if e.response is not None else ""
                raise RuntimeError(
                    f"llama-server embedder POST {path} -> {e.response.status_code}: {body}"
                ) from e
        raise RuntimeError(f"llama-server embedder POST {path} failed after retry") from last_exc

    def encode(
        self,
        texts,
        *,
        model_name = None,
        normalize = True,
    ):
        """Embed texts -> (N, dim) float32. ``model_name`` is ignored (the GGUF is
        fixed by config). Normalizes in Python to match the ST backend."""
        n = len(texts)
        if n == 0:
            return np.zeros((0, self.dim()), dtype = np.float32)
        rows: list[list[float]] = []
        batch = max(1, config.EMBED_BATCH)
        for start in range(0, n, batch):
            chunk = list(texts[start : start + batch])
            data = self._post(

View on GitHub (pinned to 203007d190)

Solutions

  1. Inspect e.__cause__ of the RuntimeError to see the underlying transport error (ConnectionReset vs ReadTimeout vs ConnectError).
  2. Check whether llama-server is being OOM-killed (dmesg / Windows Event Log) and reduce model size, n_gpu_layers, or EMBED_BATCH.
  3. Run llama-server manually and POST the same payload to reproduce the crash; if it segfaults, the GGUF or binary is bad.
  4. Catch this error at the caller and fall back to the sentence-transformers backend via RAG_EMBED_BACKEND.
  5. Reduce concurrency so multiple encode() calls do not race the single-instance restart.

Example fix

# before
vec = backend.encode([doc])  # unguarded; crash of llama-server surfaces raw

# after
try:
    vec = backend.encode([doc])
except RuntimeError as e:
    if "failed after retry" not in str(e):
        raise
    logger.warning("llama-server embedder unstable; falling back")
    vec = _st_fallback().encode([doc])
Defensive patterns

Strategy: retry

Validate before calling

import httpx

def embedder_reachable(base_url: str, timeout: float = 2.0) -> bool:
    try:
        return httpx.get(f"{base_url}/health", timeout=timeout).status_code == 200
    except (httpx.TransportError, httpx.TimeoutException):
        return False

Try / catch

for attempt in range(3):
    try:
        vecs = backend.encode(texts)
        break
    except RuntimeError as e:
        if "failed after retry" not in str(e):
            raise
        if attempt == 2:
            vecs = st_fallback_backend().encode(texts)
        else:
            backoff(2 ** attempt)

Prevention

When it happens

Trigger: Calling encode()/dim() when the llama-server process crashed (OOM-killed, segfaulted on a bad GGUF); the health endpoint passed but the server dies on the first real inference; network/socket issues on the loopback connection; a restart race where the second POST is issued before the respawned server is ready.

Common situations: VRAM/RAM exhaustion killing llama-server mid-batch; a GGUF that loads but segfaults on specific inputs; concurrent encode() calls racing the restart path; OS-level resource limits (file descriptors) on long-running workers.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/936b687433af8d15. Report an issue: GitHub.