HKUDS/DeepTutor · error · EmbeddingProviderError

Embedding provider remained rate limited after key rotation

Error message

Embedding provider remained rate limited after key rotation (Retry-After: {retry_after:g}s)

What it means

The provider returned HTTP 429 and stayed rate limited through 8 retry rounds that already rotated API keys and honored Retry-After windows. This signals a hard, persistent rate/quota condition (typically monthly quota exhaustion or an org-wide limit), not a transient sliding-window throttle.

Source

Thrown at deeptutor/services/embedding/adapters/openai_compatible.py:263

                        # 滑动窗口 429 是瞬态的：长跑（全库 reindex 数小时）里
                        # 单次 429 不该报废整跑。最多 8 轮，每轮等窗口滑过
                        # （Retry-After 优先，无头保守 60s）。月度额度耗尽的
                        # 429 会连挂 8 轮后仍然 raise，不会无限空转。
                        if rate_limit_retries < 8:
                            rate_limit_retries += 1
                            retry_after = float(response.headers.get("Retry-After", 0))
                            await asyncio.sleep(max(retry_after, 60))
                            try:
                                api_key = self._auth_api_key()
                            except RuntimeError:
                                # 池内 key 全在冷却（KeyPool 冷却 60s）。
                                # 等冷却期过后再取一次；仍取不到才认输。
                                await asyncio.sleep(65)
                                api_key = self._auth_api_key()
                            self._set_auth_header(headers, api_key)
                            continue
                        retry_after = float(response.headers.get("Retry-After", 0))
                        raise EmbeddingProviderError(
                            "Embedding provider remained rate limited after key rotation"
                            + (f" (Retry-After: {retry_after:g}s)" if retry_after else ""),
                            status=429,
                            model=model,
                            url=url,
                            provider="openai_compat",
                        )

                    if response.status_code >= 400:
                        body_text = response.text
                        if "encoding_format" not in payload and rejects_absent_encoding_format(
                            response.status_code, body_text
                        ):
                            payload["encoding_format"] = "float"
                            logger.info(
                                "Gateway requires an explicit `encoding_format`; "
                                "retrying once with 'float' (%s)",
                                url,

View on GitHub (pinned to 3e82f13042)

Solutions

  1. Check the provider dashboard: if monthly/quota cap is exhausted, top up or wait for reset — no retry logic will help
  2. Add more valid API keys to the key pool so rotation has live keys
  3. Reduce embedding batch size / add spacing between reindex batches
  4. If it is TPM throttling, lower concurrency or schedule the reindex off-peak
Defensive patterns

Strategy: retry

Validate before calling

null

Type guard

null

Try / catch

try:
    resp = await adapter.embed(req)
except EmbeddingProviderError as e:
    if e.status == 429 and "remained rate limited" in str(e):
        # persistent quota condition: pause the job, notify, do not tight-loop
        await schedule_resume_after_quota_window(e)
        return None
    raise

Prevention

When it happens

Trigger: 429 responses persisting across 8 retries spaced by max(Retry-After, 60s), with key-pool rotation exhausted — e.g. free-tier monthly quota spent, or all pooled keys simultaneously throttled.

Common situations: Long full-KB reindex jobs burning through token-per-minute limits; all keys in the pool hitting monthly caps; shared org key throttled elsewhere.

Related errors

AI-assisted analysis of HKUDS/DeepTutor@3e82f13042 (2026-08-27). Data as JSON: /api/errors/e7e0cb6a42da4fda. Report an issue: GitHub.