{"record":{"id":"e7e0cb6a42da4fda","repo":"HKUDS/DeepTutor","slug":"embedding-provider-remained-rate-limited-after-key","errorCode":null,"errorMessage":"Embedding provider remained rate limited after key rotation (Retry-After: {retry_after:g}s)","messagePattern":"Embedding provider remained rate limited after key rotation \\(Retry-After: (.+?)s\\)","errorType":"http","errorClass":"EmbeddingProviderError","httpStatus":429,"severity":"error","filePath":"deeptutor/services/embedding/adapters/openai_compatible.py","lineNumber":263,"sourceCode":"                        # 滑动窗口 429 是瞬态的：长跑（全库 reindex 数小时）里\n                        # 单次 429 不该报废整跑。最多 8 轮，每轮等窗口滑过\n                        # （Retry-After 优先，无头保守 60s）。月度额度耗尽的\n                        # 429 会连挂 8 轮后仍然 raise，不会无限空转。\n                        if rate_limit_retries < 8:\n                            rate_limit_retries += 1\n                            retry_after = float(response.headers.get(\"Retry-After\", 0))\n                            await asyncio.sleep(max(retry_after, 60))\n                            try:\n                                api_key = self._auth_api_key()\n                            except RuntimeError:\n                                # 池内 key 全在冷却（KeyPool 冷却 60s）。\n                                # 等冷却期过后再取一次；仍取不到才认输。\n                                await asyncio.sleep(65)\n                                api_key = self._auth_api_key()\n                            self._set_auth_header(headers, api_key)\n                            continue\n                        retry_after = float(response.headers.get(\"Retry-After\", 0))\n                        raise EmbeddingProviderError(\n                            \"Embedding provider remained rate limited after key rotation\"\n                            + (f\" (Retry-After: {retry_after:g}s)\" if retry_after else \"\"),\n                            status=429,\n                            model=model,\n                            url=url,\n                            provider=\"openai_compat\",\n                        )\n\n                    if response.status_code >= 400:\n                        body_text = response.text\n                        if \"encoding_format\" not in payload and rejects_absent_encoding_format(\n                            response.status_code, body_text\n                        ):\n                            payload[\"encoding_format\"] = \"float\"\n                            logger.info(\n                                \"Gateway requires an explicit `encoding_format`; \"\n                                \"retrying once with 'float' (%s)\",\n                                url,","sourceCodeStart":245,"sourceCodeEnd":281,"githubUrl":"https://github.com/HKUDS/DeepTutor/blob/3e82f130422a813cdd73c10b21a44e9325f5821a/deeptutor/services/embedding/adapters/openai_compatible.py#L245-L281","documentation":"The provider returned HTTP 429 and stayed rate limited through 8 retry rounds that already rotated API keys and honored Retry-After windows. This signals a hard, persistent rate/quota condition (typically monthly quota exhaustion or an org-wide limit), not a transient sliding-window throttle.","triggerScenarios":"429 responses persisting across 8 retries spaced by max(Retry-After, 60s), with key-pool rotation exhausted — e.g. free-tier monthly quota spent, or all pooled keys simultaneously throttled.","commonSituations":"Long full-KB reindex jobs burning through token-per-minute limits; all keys in the pool hitting monthly caps; shared org key throttled elsewhere.","solutions":["Check the provider dashboard: if monthly/quota cap is exhausted, top up or wait for reset — no retry logic will help","Add more valid API keys to the key pool so rotation has live keys","Reduce embedding batch size / add spacing between reindex batches","If it is TPM throttling, lower concurrency or schedule the reindex off-peak"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"null","typeGuard":"null","tryCatchPattern":"try:\n    resp = await adapter.embed(req)\nexcept EmbeddingProviderError as e:\n    if e.status == 429 and \"remained rate limited\" in str(e):\n        # persistent quota condition: pause the job, notify, do not tight-loop\n        await schedule_resume_after_quota_window(e)\n        return None\n    raise","preventionTips":["Pool multiple API keys so rotation has live keys","Chunk large reindex jobs and respect provider TPM limits","Monitor quota dashboards before multi-hour embedding runs"],"tags":["rate-limit","http-429","quota","retries-exhausted","embeddings"],"backgroundTag":"rate-limit-exhausted","analyzedSha":"3e82f130422a813cdd73c10b21a44e9325f5821a","analyzedAt":"2026-08-27T06:57:25.364Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}