What causes "Ollama API failed after {self._max_retries} retries: {last_error}" in headroomlabs-ai/headroom?

Confirm the daemon: curl http://localhost:11434/api/tags from the same host/network namespace as your app.

headroomlabs-ai/headroom · error · ConnectionError

Ollama API failed after {self._max_retries} retries: {last_e

Error message

Ollama API failed after {self._max_retries} retries: {last_error}

What it means

Raised after OllamaEmbedder exhausts self._max_retries (default 3) attempts. Only httpx.ConnectError, TimeoutException, and HTTPStatusError are retried with backoff (0.5s base); this ConnectionError reports last_error and means the Ollama server stayed unreachable or slow across every attempt.

Source

Thrown at headroom/memory/adapters/embedders.py:935

                    self._detected_dimension = len(embedding)

                return embedding

            except (httpx.ConnectError, httpx.TimeoutException, httpx.HTTPStatusError) as e:
                last_error = e
                delay = self.RETRY_DELAY_BASE * (2**attempt)
                logger.warning(
                    f"Ollama API error (attempt {attempt + 1}/{self._max_retries}): {e}. "
                    f"Retrying in {delay:.1f}s..."
                )
                await asyncio.sleep(delay)

            except Exception as e:
                # Non-retryable error
                raise ConnectionError(f"Ollama API error: {e}") from e

        # All retries exhausted
        raise ConnectionError(
            f"Ollama API failed after {self._max_retries} retries: {last_error}"
        ) from last_error

    async def embed(self, text: str) -> np.ndarray:
        """Generate an embedding for a single text.

        Args:
            text: The text to embed.

        Returns:
            Normalized embedding vector as float32 numpy array.

        Raises:
            ConnectionError: If API call fails after retries.
        """
        # Handle empty string
        if not text or not text.strip():
            return np.zeros(self.dimension, dtype=np.float32)

View on GitHub (pinned to 322425c43b)

Solutions

Confirm the daemon: curl http://localhost:11434/api/tags from the same host/network namespace as your app.
Start it: `ollama serve` (or restart the docker container with correct port mapping / host.docker.internal).
Pre-warm the model once (`curl /api/embed` or a tiny embed call) so later calls don't pay the model-load timeout.
Increase tolerance: OllamaEmbedder(max_retries=5) or a larger timeout if cold starts are slow.
Fix base_url to the address the daemon actually binds (it defaults to localhost).

Example fix

# before
emb = OllamaEmbedder()  # ConnectionError: failed after 3 retries — server not running

# after (shell)
ollama serve &
curl -s http://localhost:11434/api/tags  # verify, then rerun app

Defensive patterns

Strategy: retry

Validate before calling

import httpx, os

def ollama_up(base_url: str = "http://localhost:11434") -> bool:
    try:
        return httpx.get(f"{base_url}/api/tags", timeout=3).status_code == 200
    except httpx.HTTPError:
        return False

if not ollama_up(os.environ.get("OLLAMA_BASE_URL", "http://localhost:11434")):
    raise SystemExit("Ollama daemon unreachable; start it with `ollama serve`")

Try / catch

async def embed_retry(emb, text, tries=3):
    for i in range(tries):
        try:
            return await emb.embed(text)
        except ConnectionError as e:
            if "failed after" in str(e) and i + 1 < tries:
                await asyncio.sleep(10 * (i + 1))
                continue
            raise

Prevention

Health-check /api/tags at startup before accepting embedding work.
Use OllamaEmbedder(max_retries=5) plus KEEP_ALIVE settings to survive cold model loads.
In containers, point base_url at host.docker.internal (or the service name), not localhost.

When it happens

Trigger: Calling embed()/embed_batch() while the Ollama daemon is down (`ollama serve` not running), listening on a different host/port than base_url, blocked by firewall, or so overloaded (model loading, GPU contention) that every request times out.

Common situations: Ollama not started / crashed from OOM; base_url pointing at a remote machine that's off; first-call cold start loading a large model exceeding REQUEST_TIMEOUT each time; docker networking so localhost:11434 inside a container doesn't reach the host daemon.

Related errors

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/ff093be3c871639d. Report an issue: GitHub.