{"record":{"id":"396e80563da99c15","repo":"headroomlabs-ai/headroom","slug":"openai-api-failed-after-self-max-retries-retrie","errorCode":null,"errorMessage":"OpenAI API failed after {self._max_retries} retries: {last_error}","messagePattern":"OpenAI API failed after (.+?) retries: (.+?)","errorType":"exception","errorClass":"ConnectionError","httpStatus":null,"severity":"error","filePath":"headroom/memory/adapters/embedders.py","lineNumber":702,"sourceCode":"                # Extract embeddings in order\n                embeddings = [np.array(item.embedding, dtype=np.float32) for item in response.data]\n                return embeddings\n\n            except (APIConnectionError, APITimeoutError, RateLimitError) as e:\n                last_error = e\n                delay = self.RETRY_DELAY_BASE * (2**attempt)\n                logger.warning(\n                    f\"OpenAI API error (attempt {attempt + 1}/{self._max_retries}): {e}. \"\n                    f\"Retrying in {delay:.1f}s...\"\n                )\n                await asyncio.sleep(delay)\n\n            except Exception as e:\n                # Non-retryable error\n                raise ConnectionError(f\"OpenAI API error: {e}\") from e\n\n        # All retries exhausted\n        raise ConnectionError(\n            f\"OpenAI API failed after {self._max_retries} retries: {last_error}\"\n        ) from last_error\n\n    async def embed(self, text: str) -> np.ndarray:\n        \"\"\"Generate an embedding for a single text.\n\n        Args:\n            text: The text to embed.\n\n        Returns:\n            Normalized embedding vector as float32 numpy array.\n\n        Raises:\n            ConnectionError: If API call fails after retries.\n        \"\"\"\n        # Handle empty string\n        if not text or not text.strip():\n            return np.zeros(self.dimension, dtype=np.float32)","sourceCodeStart":684,"sourceCodeEnd":720,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/memory/adapters/embedders.py#L684-L720","documentation":"Raised after OpenAIEmbedder's retry loop exhausts self._max_retries (default 3) attempts. Only transient errors — APIConnectionError, APITimeoutError, RateLimitError — are retried with exponential backoff (1s, 2s, 4s by default); this ConnectionError reports the last_error and means connectivity or rate limits persisted through every attempt.","triggerScenarios":"Calling embed()/embed_batch() during an outage, through a dead proxy, under sustained 429 rate limiting, or with DNS failure — each attempt fails with a retryable error until the budget is spent.","commonSituations":"Free-tier keys hitting embedding rate limits in batch jobs; regional API outage; corporate firewall blocking api.openai.com; flaky mobile tethers; retry budget too small for how long the incident lasts.","solutions":["Wait for rate-limit windows to reset or upgrade the account's tier; the backoff (max ~3 tries) may be shorter than the 429 window.","Increase the budget at construction: OpenAIEmbedder(max_retries=6) so backoff reaches ~32s.","Batch less aggressively — add spacing between embed_batch() calls to stay under RPM limits.","Check connectivity (curl https://api.openai.com/v1/models) and proxy/firewall rules if errors are connection-shaped.","Fall back to LocalEmbedder/OllamaEmbedder for offline resilience."],"exampleFix":"# before\nemb = OpenAIEmbedder()  # default 3 retries, fails during rate limiting\n\n# after\nemb = OpenAIEmbedder(max_retries=6)  # backoff reaches ~32s, survives longer 429 windows","handlingStrategy":"retry","validationCode":"import os\n\n# Right-size the retry budget before a batch job instead of the default 3.\nbatch_size = int(os.environ.get(\"EMBED_BATCH\", \"100\"))\nembedder = OpenAIEmbedder(max_retries=6 if batch_size > 50 else 3)","typeGuard":null,"tryCatchPattern":"async def embed_with_backoff(emb, text, outer_tries=3):\n    for i in range(outer_tries):\n        try:\n            return await emb.embed(text)\n        except ConnectionError as e:\n            if \"failed after\" in str(e) and i + 1 < outer_tries:\n                await asyncio.sleep(30 * (i + 1))  # outer backoff beyond inner retries\n                continue\n            raise","preventionTips":["Raise max_retries (construction arg) when running large batches against rate-limited tiers.","Pace batch calls — sleep between embed_batch() calls to stay under RPM.","Add an outer retry with minutes-scale backoff; the inner one only reaches ~7s total."],"tags":["api","openai","retry","rate-limit","embeddings","network"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}