headroomlabs-ai/headroom · warning · HTTPException

Rate limited. Retry after {wait_seconds:.1f}s

Error message

Rate limited. Retry after {wait_seconds:.1f}s

What it means

The Gemini handler applies its own rate limit keyed by the inbound x-goog-api-key header (first 20 chars) before converting the request. On exhaustion it records a rate_limited metric and raises HTTPException 429 with detail 'Rate limited. Retry after Ns'. Unlike the Anthropic handler it does not set a Retry-After header and does not hold a pre-upstream semaphore at this point, so the parse must come from the detail message.

Source

Thrown at headroom/proxy/handlers/gemini.py:398

        from headroom.proxy.helpers import get_memory_injection_mode, log_memory_injection
        from headroom.proxy.memory_decision import MemoryDecision
        from headroom.proxy.memory_query import MemoryQuery

        memory_decision = MemoryDecision.decide(
            headers=request.headers,
            memory_handler=self.memory_handler,
            memory_user_id=memory_user_id,
            mode_name=get_memory_injection_mode(),
        )
        memory_decision.apply_to_tags(tags)

        # Rate limiting (use Gemini API key)
        if self.rate_limiter:
            rate_key = headers.get("x-goog-api-key", "default")[:20]
            allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
            if not allowed:
                await self.metrics.record_rate_limited(provider=provider_name)
                raise HTTPException(
                    status_code=429,
                    detail=f"Rate limited. Retry after {wait_seconds:.1f}s",
                )

        # Convert Gemini format to messages for optimization
        system_instruction = body.get("systemInstruction")
        messages, preserved_indices = self._gemini_contents_to_messages(
            contents, system_instruction
        )

        # Store original content entries that have non-text parts before compression
        preserved_contents = {idx: contents[idx] for idx in preserved_indices}

        # Early exit if ALL content has non-text parts (nothing to compress)
        if len(preserved_indices) == len(contents):
            # All content has non-text parts, skip compression entirely
            # Just forward the request as-is
            query_params = dict(request.query_params)

View on GitHub (pinned to 322425c43b)

Solutions

  1. Parse the wait from the 429 detail text and back off at least that long before retrying.
  2. Raise the limiter's allowance for Gemini keys, or distribute clients across multiple keys so buckets don't collide.
  3. Add client-side request pacing/concurrency limits sized to the configured window.

Example fix

# before
for attempt in range(5):
    resp = post_gemini(body)  # immediate retries keep getting 429

# after
resp = post_gemini(body)
if resp.status_code == 429:
    wait = float(re.search(r"after ([0-9.]+)s", resp.json()["detail"]).group(1))
    time.sleep(wait + 0.5)
Defensive patterns

Strategy: retry

Validate before calling

# No pre-check available; pace requests to the configured per-key window before sending.

Try / catch

resp = await client.post(gemini_url, json=body)
if resp.status_code == 429:
    m = re.search(r"after ([0-9.]+)s", resp.json()["detail"])
    await asyncio.sleep(float(m.group(1)) + 0.5 if m else 5.0)
    resp = await client.post(gemini_url, json=body)

Prevention

When it happens

Trigger: Sending more Gemini-format requests (x-goog-api-key auth) within the window than the limiter allows for that key prefix; multiple clients sharing one Google API key through the proxy.

Common situations: Several agents or team members configured with the same Gemini key funneled through one proxy; a client retry loop that ignores 429; per-key limit set lower than the aggregate fan-out.

Related errors

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/acec5e18f2fc912a. Report an issue: GitHub.