{"record":{"id":"b1ede388be681c42","repo":"headroomlabs-ai/headroom","slug":"token-endpoint-unreachable-e","errorCode":null,"errorMessage":"token endpoint unreachable: {e}","messagePattern":"token endpoint unreachable: (.+?)","errorType":"exception","errorClass":"OAuth2Error","httpStatus":null,"severity":"error","filePath":"plugins/headroom-oauth2/src/headroom_oauth2/provider.py","lineNumber":136,"sourceCode":"            headers[\"Authorization\"] = \"Basic \" + creds\n        else:\n            form[\"client_id\"] = self.client_id\n            form[\"client_secret\"] = self.client_secret\n        req = urllib.request.Request(\n            self.token_url,\n            data=urllib.parse.urlencode(form).encode(),\n            headers=headers,\n            method=\"POST\",\n        )\n        try:\n            with urllib.request.urlopen(req, timeout=self.timeout) as resp:\n                payload = json.load(resp)\n        except HTTPError as e:\n            with suppress(Exception):\n                e.read()  # drain; do NOT surface the IdP body (may echo sensitive context)\n            raise OAuth2Error(f\"token endpoint returned HTTP {e.code}\") from None\n        except (URLError, OSError) as e:\n            raise OAuth2Error(f\"token endpoint unreachable: {e}\") from None\n        except json.JSONDecodeError:\n            raise OAuth2Error(\"token endpoint returned non-JSON\") from None\n        token = payload.get(\"access_token\")\n        if not token:\n            raise OAuth2Error(\"token endpoint response had no access_token\")\n        raw = payload.get(\"expires_in\")\n        try:\n            ttl = int(float(raw))  # tolerate \"3600\", \"3600.0\", 3600, or a JSON float\n        except (TypeError, ValueError):\n            ttl = 300\n        ttl = max(1, ttl)  # 0/negative would cause a stale token or per-request minting\n        log.info(\"oauth2: minted token (ttl=%ss, scopes=%s)\", ttl, self.scopes or \"-\")\n        return token, ttl\n","sourceCodeStart":118,"sourceCodeEnd":150,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/plugins/headroom-oauth2/src/headroom_oauth2/provider.py#L118-L150","documentation":"The token request failed before getting an HTTP response — `URLError` or `OSError` from stdlib urllib — and is wrapped as OAuth2Error with the underlying reason. This means DNS failure, connection refused, TLS handshake failure, or the configured `timeout_seconds` (default 30s) elapsing; the IdP endpoint is unreachable from this process.","triggerScenarios":"DNS for the IdP host not resolvable from inside the container; firewall/NetworkPolicy blocking egress on 443; IdP down or restarting; a proxy required by the network but not configured for urllib; slow IdP exceeding timeout_seconds.","commonSituations":"Kubernetes pods without egress NetworkPolicy allowances; corporate networks requiring an HTTP proxy that Python's urllib only honors via HTTPS_PROXY env; staging stacks starting up while the IdP container is not ready; transient IdP outages or load-balancer health-check gaps.","solutions":["Verify reachability from the same environment: `curl -v $TOKEN_URL` (or `python -c \"import urllib.request; urllib.request.urlopen('$TOKEN_URL')\"`) to see DNS/connect/TLS errors with full context","Configure the required proxy via `HTTPS_PROXY`/`HTTP_PROXY` env vars if the network mandates one (urllib honors them)","For transient/startup races, retry with backoff around token minting (the provider caches tokens, so wrap the first call), and raise `timeout_seconds` if the IdP is genuinely slow"],"exampleFix":"# before\nprovider = OAuth2ClientCredentials(token_url=..., timeout_seconds=30.0, ...)\ntoken = provider.get_token()  # OAuth2Error: token endpoint unreachable: ...\n\n# after\nimport os, time\nos.environ.setdefault(\"HTTPS_PROXY\", \"http://proxy.corp:3128\")  # if network requires it\nfor attempt in range(3):\n    try:\n        token = provider.get_token(); break\n    except OAuth2Error:\n        if attempt == 2: raise\n        time.sleep(2 ** attempt)","handlingStrategy":"retry","validationCode":"import socket\nfrom urllib.parse import urlparse\n\ndef token_endpoint_reachable(url: str, timeout: float = 3.0) -> bool:\n    p = urlparse(url)\n    try:\n        socket.create_connection((p.hostname, p.port or 443), timeout=timeout).close()\n        return True\n    except OSError:\n        return False\n\nif not token_endpoint_reachable(url):\n    raise RuntimeError(f\"IdP unreachable from this pod: {url}\")","typeGuard":null,"tryCatchPattern":"from headroom_oauth2.provider import OAuth2Error\n\nfor attempt in range(4):\n    try:\n        token = provider.get_token()\n        break\n    except OAuth2Error as e:\n        if \"unreachable\" not in str(e) or attempt == 3:\n            raise\n        time.sleep(1.5 ** attempt)  # DNS/connect/timeout — transient by nature","preventionTips":["Verify egress (DNS, firewall, NetworkPolicy) from the exact runtime, not your laptop","Set HTTPS_PROXY/HTTP_PROXY when the network requires an outbound proxy — urllib honors them","Raise timeout_seconds for slow IdPs; keep retry-with-backoff around first token mint since tokens are cached"],"tags":["oauth2","network","dns","timeout","retryable"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}