headroomlabs-ai/headroom · error · ConnectionError

OpenAI API failed after {self._max_retries} retries: {last_e

Error message

OpenAI API failed after {self._max_retries} retries: {last_error}

What it means

Raised after OpenAIEmbedder's retry loop exhausts self._max_retries (default 3) attempts. Only transient errors — APIConnectionError, APITimeoutError, RateLimitError — are retried with exponential backoff (1s, 2s, 4s by default); this ConnectionError reports the last_error and means connectivity or rate limits persisted through every attempt.

Source

Thrown at headroom/memory/adapters/embedders.py:702

                # Extract embeddings in order
                embeddings = [np.array(item.embedding, dtype=np.float32) for item in response.data]
                return embeddings

            except (APIConnectionError, APITimeoutError, RateLimitError) as e:
                last_error = e
                delay = self.RETRY_DELAY_BASE * (2**attempt)
                logger.warning(
                    f"OpenAI API error (attempt {attempt + 1}/{self._max_retries}): {e}. "
                    f"Retrying in {delay:.1f}s..."
                )
                await asyncio.sleep(delay)

            except Exception as e:
                # Non-retryable error
                raise ConnectionError(f"OpenAI API error: {e}") from e

        # All retries exhausted
        raise ConnectionError(
            f"OpenAI API failed after {self._max_retries} retries: {last_error}"
        ) from last_error

    async def embed(self, text: str) -> np.ndarray:
        """Generate an embedding for a single text.

        Args:
            text: The text to embed.

        Returns:
            Normalized embedding vector as float32 numpy array.

        Raises:
            ConnectionError: If API call fails after retries.
        """
        # Handle empty string
        if not text or not text.strip():
            return np.zeros(self.dimension, dtype=np.float32)

View on GitHub (pinned to 322425c43b)

Solutions

  1. Wait for rate-limit windows to reset or upgrade the account's tier; the backoff (max ~3 tries) may be shorter than the 429 window.
  2. Increase the budget at construction: OpenAIEmbedder(max_retries=6) so backoff reaches ~32s.
  3. Batch less aggressively — add spacing between embed_batch() calls to stay under RPM limits.
  4. Check connectivity (curl https://api.openai.com/v1/models) and proxy/firewall rules if errors are connection-shaped.
  5. Fall back to LocalEmbedder/OllamaEmbedder for offline resilience.

Example fix

# before
emb = OpenAIEmbedder()  # default 3 retries, fails during rate limiting

# after
emb = OpenAIEmbedder(max_retries=6)  # backoff reaches ~32s, survives longer 429 windows
Defensive patterns

Strategy: retry

Validate before calling

import os

# Right-size the retry budget before a batch job instead of the default 3.
batch_size = int(os.environ.get("EMBED_BATCH", "100"))
embedder = OpenAIEmbedder(max_retries=6 if batch_size > 50 else 3)

Try / catch

async def embed_with_backoff(emb, text, outer_tries=3):
    for i in range(outer_tries):
        try:
            return await emb.embed(text)
        except ConnectionError as e:
            if "failed after" in str(e) and i + 1 < outer_tries:
                await asyncio.sleep(30 * (i + 1))  # outer backoff beyond inner retries
                continue
            raise

Prevention

When it happens

Trigger: Calling embed()/embed_batch() during an outage, through a dead proxy, under sustained 429 rate limiting, or with DNS failure — each attempt fails with a retryable error until the budget is spent.

Common situations: Free-tier keys hitting embedding rate limits in batch jobs; regional API outage; corporate firewall blocking api.openai.com; flaky mobile tethers; retry budget too small for how long the incident lasts.

Related errors

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/396e80563da99c15. Report an issue: GitHub.