headroomlabs-ai/headroom · error · OAuth2Error

token endpoint unreachable: {e}

Error message

token endpoint unreachable: {e}

What it means

The token request failed before getting an HTTP response — `URLError` or `OSError` from stdlib urllib — and is wrapped as OAuth2Error with the underlying reason. This means DNS failure, connection refused, TLS handshake failure, or the configured `timeout_seconds` (default 30s) elapsing; the IdP endpoint is unreachable from this process.

Source

Thrown at plugins/headroom-oauth2/src/headroom_oauth2/provider.py:136

            headers["Authorization"] = "Basic " + creds
        else:
            form["client_id"] = self.client_id
            form["client_secret"] = self.client_secret
        req = urllib.request.Request(
            self.token_url,
            data=urllib.parse.urlencode(form).encode(),
            headers=headers,
            method="POST",
        )
        try:
            with urllib.request.urlopen(req, timeout=self.timeout) as resp:
                payload = json.load(resp)
        except HTTPError as e:
            with suppress(Exception):
                e.read()  # drain; do NOT surface the IdP body (may echo sensitive context)
            raise OAuth2Error(f"token endpoint returned HTTP {e.code}") from None
        except (URLError, OSError) as e:
            raise OAuth2Error(f"token endpoint unreachable: {e}") from None
        except json.JSONDecodeError:
            raise OAuth2Error("token endpoint returned non-JSON") from None
        token = payload.get("access_token")
        if not token:
            raise OAuth2Error("token endpoint response had no access_token")
        raw = payload.get("expires_in")
        try:
            ttl = int(float(raw))  # tolerate "3600", "3600.0", 3600, or a JSON float
        except (TypeError, ValueError):
            ttl = 300
        ttl = max(1, ttl)  # 0/negative would cause a stale token or per-request minting
        log.info("oauth2: minted token (ttl=%ss, scopes=%s)", ttl, self.scopes or "-")
        return token, ttl

View on GitHub (pinned to 322425c43b)

Solutions

1. Verify reachability from the same environment: `curl -v $TOKEN_URL` (or `python -c "import urllib.request; urllib.request.urlopen('$TOKEN_URL')"`) to see DNS/connect/TLS errors with full context
2. Configure the required proxy via `HTTPS_PROXY`/`HTTP_PROXY` env vars if the network mandates one (urllib honors them)
3. For transient/startup races, retry with backoff around token minting (the provider caches tokens, so wrap the first call), and raise `timeout_seconds` if the IdP is genuinely slow

Example fix

# before
provider = OAuth2ClientCredentials(token_url=..., timeout_seconds=30.0, ...)
token = provider.get_token()  # OAuth2Error: token endpoint unreachable: ...

# after
import os, time
os.environ.setdefault("HTTPS_PROXY", "http://proxy.corp:3128")  # if network requires it
for attempt in range(3):
    try:
        token = provider.get_token(); break
    except OAuth2Error:
        if attempt == 2: raise
        time.sleep(2 ** attempt)

import socket
from urllib.parse import urlparse

def token_endpoint_reachable(url: str, timeout: float = 3.0) -> bool:
    p = urlparse(url)
    try:
        socket.create_connection((p.hostname, p.port or 443), timeout=timeout).close()
        return True
    except OSError:
        return False

if not token_endpoint_reachable(url):
    raise RuntimeError(f"IdP unreachable from this pod: {url}")

from headroom_oauth2.provider import OAuth2Error

for attempt in range(4):
    try:
        token = provider.get_token()
        break
    except OAuth2Error as e:
        if "unreachable" not in str(e) or attempt == 3:
            raise
        time.sleep(1.5 ** attempt)  # DNS/connect/timeout — transient by nature

When it happens

Trigger: DNS for the IdP host not resolvable from inside the container; firewall/NetworkPolicy blocking egress on 443; IdP down or restarting; a proxy required by the network but not configured for urllib; slow IdP exceeding timeout_seconds.

Common situations: Kubernetes pods without egress NetworkPolicy allowances; corporate networks requiring an HTTP proxy that Python's urllib only honors via HTTPS_PROXY env; staging stacks starting up while the IdP container is not ready; transient IdP outages or load-balancer health-check gaps.

Related errors

• upstream request failed: {0}
• `{' '.join(cmd)}` produced no output for {idle_cap}s. Check
• headroom proxy was not ready at {url}: {last_error}
• tiktoken encoding {encoding_name!r} previously failed to loa
• tiktoken encoding {encoding_name!r} load timed out

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/b1ede388be681c42. Report an issue: GitHub.