{"record":{"id":"a42c7f8c2fb16184","repo":"headroomlabs-ai/headroom","slug":"headroom-proxy-was-not-ready-at-url-last-error","errorCode":null,"errorMessage":"headroom proxy was not ready at {url}: {last_error}","messagePattern":"headroom proxy was not ready at (.+?): (.+?)","errorType":"exception","errorClass":"TimeoutError","httpStatus":null,"severity":"error","filePath":"headroom/testing/harness.py","lineNumber":1507,"sourceCode":"            self._log_file.close()\n            self._log_file = None\n\n    def _wait_ready(self) -> None:\n        assert self._process is not None\n        url = f\"http://127.0.0.1:{self.port}{self.ready_path}\"\n        deadline = time.monotonic() + self.timeout_s\n        last_error: str | None = None\n        while time.monotonic() < deadline:\n            if self._process.poll() is not None:\n                raise RuntimeError(f\"headroom proxy exited with code {self._process.returncode}\")\n            try:\n                with urllib.request.urlopen(url, timeout=0.5) as response:\n                    if response.status == 200:\n                        return\n            except (OSError, urllib.error.URLError) as exc:\n                last_error = str(exc)\n            time.sleep(0.25)\n        raise TimeoutError(f\"headroom proxy was not ready at {url}: {last_error}\")\n\n\nclass _NoopClient:\n    \"\"\"Minimal original-client object for SDK simulations without upstream I/O.\"\"\"\n\n\nclass ScenarioOrchestrator:\n    \"\"\"Runs built scenarios against local no-key tasks and evaluates guarantees.\"\"\"\n\n    def __init__(\n        self,\n        scenarios: Sequence[HarnessScenario],\n        *,\n        guarantees: Sequence[Guarantee] = DEFAULT_GUARANTEES,\n    ) -> None:\n        if not scenarios:\n            raise ValueError(\"ScenarioOrchestrator requires at least one scenario\")\n        self._scenarios = tuple(scenarios)","sourceCodeStart":1489,"sourceCodeEnd":1525,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/testing/harness.py#L1489-L1525","documentation":"Raised by _wait_ready() when the headroom proxy subprocess is still alive but did not answer HTTP 200 on http://127.0.0.1:{port}{ready_path} within timeout_s seconds. The last connection error is embedded in the message. It indicates a slow or hung startup (often a blocking model/vocab download or slow import), not a crash.","triggerScenarios":"First run downloads tokenizer vocabularies or model metadata synchronously at proxy startup; ready_path configured to an endpoint the proxy doesn't expose; a firewall/proxy env var (HTTP_PROXY) intercepting the loopback urllib request; timeout_s left at default on a cold CI machine.","commonSituations":"Cold CI containers with no warm caches and restricted egress; corporate proxies env vars making urllib.request.urlopen loopback calls fail; a readiness route renamed between proxy versions; slow disk/network for tiktoken cache.","solutions":["Increase the startup budget: pass a larger timeout_s to the proxy harness.","Verify ready_path matches an endpoint the proxy actually serves 200 on (e.g. /healthz).","Exclude loopback from proxy env: NO_PROXY=127.0.0.1, or unset HTTP_PROXY/HTTPS_PROXY in the harness environment.","Warm caches (e.g. TIKTOKEN_CACHE_DIR pre-populated) so the proxy starts fast."],"exampleFix":"# before\nproxy = HeadroomProxy(command=cmd, port=8080, ready_path=\"/healthz\")\nproxy.start()  # TimeoutError\n\n# after\nimport os\nos.environ[\"NO_PROXY\"] = \"127.0.0.1,localhost\"\nproxy = HeadroomProxy(command=cmd, port=8080, ready_path=\"/healthz\", timeout_s=60.0)\nproxy.start()","handlingStrategy":"retry","validationCode":"import socket, urllib.request\nurl = f\"http://127.0.0.1:{port}{ready_path}\"\n# pre-check loopback reachability and that no HTTP_PROXY intercepts it\nassert not urllib.request.getproxies().get(\"http\") or \"127.0.0.1\" in os.environ.get(\"NO_PROXY\", \"\")","typeGuard":null,"tryCatchPattern":"for attempt in range(3):\n    try:\n        proxy.start()\n        break\n    except TimeoutError as e:\n        if attempt == 2:\n            raise\n        proxy.timeout_s *= 2  # back off and retry with a longer budget","preventionTips":["Set NO_PROXY=127.0.0.1,localhost in proxied environments.","Use a generous timeout_s on cold CI.","Warm tokenizer caches in the image so proxy startup is fast.","Confirm ready_path matches the proxy's health endpoint."],"tags":["timeout","proxy","startup","network","ci"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}