unslothai/unsloth · error · RuntimeError

llama-server embedder POST {path} -> {e.response.status_code

Error message

llama-server embedder POST {path} -> {e.response.status_code}: {body}

What it means

Raised when a POST to the llama-server embedder (e.g. /v1/embeddings) returns a non-2xx HTTP status. Unlike transport errors, HTTP status errors are not retried — the backend immediately converts the httpx.HTTPStatusError into a RuntimeError carrying the status code and up to 500 chars of the response body. It usually indicates a request the running server cannot serve, such as a context-length overflow or malformed JSON payload.

Source

Thrown at studio/backend/core/rag/embed_llama_server.py:725

    def _post(self, path: str, payload: dict) -> dict:
        """POST to the server, restarting once and retrying on a dropped connection
        (the reaper may have killed us) or a timeout (the bundled build sometimes
        wedges a request); a fresh server unsticks both."""
        last_exc: Exception | None = None
        for attempt in range(2):
            self._ensure_ready()
            try:
                resp = self._client.post(f"{self._base_url}{path}", json = payload)
                resp.raise_for_status()
                return resp.json()
            except (*_TRANSPORT_ERRORS, httpx.TimeoutException) as e:
                last_exc = e
                if attempt == 0:
                    self._restart()
                    continue
            except httpx.HTTPStatusError as e:
                body = e.response.text[:500] if e.response is not None else ""
                raise RuntimeError(
                    f"llama-server embedder POST {path} -> {e.response.status_code}: {body}"
                ) from e
        raise RuntimeError(f"llama-server embedder POST {path} failed after retry") from last_exc

    def encode(
        self,
        texts,
        *,
        model_name = None,
        normalize = True,
    ):
        """Embed texts -> (N, dim) float32. ``model_name`` is ignored (the GGUF is
        fixed by config). Normalizes in Python to match the ST backend."""
        n = len(texts)
        if n == 0:
            return np.zeros((0, self.dim()), dtype = np.float32)
        rows: list[list[float]] = []
        batch = max(1, config.EMBED_BATCH)

View on GitHub (pinned to 203007d190)

Solutions

1. Read the status code and body snippet in the message — llama-server states the exact reason (e.g. 'prompt too long', 'model not found').
2. Lower config.EMBED_BATCH so one batch's token count fits within the server's context size.
3. Pre-truncate/normalize input texts before encode() so no single text exceeds the context budget.
4. Ensure the llama-server subprocess was started with a --ctx-size at least as large as batch_size * max_text_tokens.
5. Restart the backend (it re-spawns the server) if the server has drifted into a bad state.

Example fix

# before
batch = max(1, config.EMBED_BATCH)  # e.g. 512 texts at once overflows ctx

# after
batch = max(1, min(config.EMBED_BATCH, 64))  # keep batch token footprint well under ctx-size

from core import config

MAX_TOKENS_PER_TEXT = 8192  # keep well under server ctx-size

def batch_fits_context(texts: list[str]) -> bool:
    per_call = max(1, config.EMBED_BATCH)
    worst = max((len(t) for t in texts), default=0)
    return per_call * (worst // 3 + 1) < 8192  # ~3 chars/token heuristic

try:
    data = backend._post("/v1/embeddings", payload)
except RuntimeError as e:
    if "-> 4" in str(e) or "-> 5" in str(e):  # 4xx/5xx surfaced
        log.error("embedder rejected batch: %s", e)
        raise EmbeddingRequestError(str(e)) from e
    raise

When it happens

Trigger: Calling encode() with a batch whose combined prompt exceeds the server's context window (400/413); posting to /v1/embeddings after the server was restarted with a different model so the 'embedding' model name or input shape is rejected (404/400); server under memory pressure returning 500.

Common situations: EMBED_BATCH set larger than the server's --ctx-size divided by typical chunk length; a chunking change that produced very long texts; version skew between the client's request format and the llama-server build; server restarted mid-session with different flags.

Related errors

• llama-server embedder failed to become healthy. Last output:
• llama-server embedder POST {path} failed after retry
• embedder returned {len(items)} vectors for {len(chunk)} inpu
• family {getattr(fam, 'name', fam)!r} declares no prequant_fi
• lockfile not found: {path}

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/f11de53af0335abb. Report an issue: GitHub.