headroomlabs-ai/headroom · error · TimeoutError

headroom proxy was not ready at {url}: {last_error}

Error message

headroom proxy was not ready at {url}: {last_error}

What it means

Raised by _wait_ready() when the headroom proxy subprocess is still alive but did not answer HTTP 200 on http://127.0.0.1:{port}{ready_path} within timeout_s seconds. The last connection error is embedded in the message. It indicates a slow or hung startup (often a blocking model/vocab download or slow import), not a crash.

Source

Thrown at headroom/testing/harness.py:1507

            self._log_file.close()
            self._log_file = None

    def _wait_ready(self) -> None:
        assert self._process is not None
        url = f"http://127.0.0.1:{self.port}{self.ready_path}"
        deadline = time.monotonic() + self.timeout_s
        last_error: str | None = None
        while time.monotonic() < deadline:
            if self._process.poll() is not None:
                raise RuntimeError(f"headroom proxy exited with code {self._process.returncode}")
            try:
                with urllib.request.urlopen(url, timeout=0.5) as response:
                    if response.status == 200:
                        return
            except (OSError, urllib.error.URLError) as exc:
                last_error = str(exc)
            time.sleep(0.25)
        raise TimeoutError(f"headroom proxy was not ready at {url}: {last_error}")


class _NoopClient:
    """Minimal original-client object for SDK simulations without upstream I/O."""


class ScenarioOrchestrator:
    """Runs built scenarios against local no-key tasks and evaluates guarantees."""

    def __init__(
        self,
        scenarios: Sequence[HarnessScenario],
        *,
        guarantees: Sequence[Guarantee] = DEFAULT_GUARANTEES,
    ) -> None:
        if not scenarios:
            raise ValueError("ScenarioOrchestrator requires at least one scenario")
        self._scenarios = tuple(scenarios)

View on GitHub (pinned to 322425c43b)

Solutions

  1. Increase the startup budget: pass a larger timeout_s to the proxy harness.
  2. Verify ready_path matches an endpoint the proxy actually serves 200 on (e.g. /healthz).
  3. Exclude loopback from proxy env: NO_PROXY=127.0.0.1, or unset HTTP_PROXY/HTTPS_PROXY in the harness environment.
  4. Warm caches (e.g. TIKTOKEN_CACHE_DIR pre-populated) so the proxy starts fast.

Example fix

# before
proxy = HeadroomProxy(command=cmd, port=8080, ready_path="/healthz")
proxy.start()  # TimeoutError

# after
import os
os.environ["NO_PROXY"] = "127.0.0.1,localhost"
proxy = HeadroomProxy(command=cmd, port=8080, ready_path="/healthz", timeout_s=60.0)
proxy.start()
Defensive patterns

Strategy: retry

Validate before calling

import socket, urllib.request
url = f"http://127.0.0.1:{port}{ready_path}"
# pre-check loopback reachability and that no HTTP_PROXY intercepts it
assert not urllib.request.getproxies().get("http") or "127.0.0.1" in os.environ.get("NO_PROXY", "")

Try / catch

for attempt in range(3):
    try:
        proxy.start()
        break
    except TimeoutError as e:
        if attempt == 2:
            raise
        proxy.timeout_s *= 2  # back off and retry with a longer budget

Prevention

When it happens

Trigger: First run downloads tokenizer vocabularies or model metadata synchronously at proxy startup; ready_path configured to an endpoint the proxy doesn't expose; a firewall/proxy env var (HTTP_PROXY) intercepting the loopback urllib request; timeout_s left at default on a cold CI machine.

Common situations: Cold CI containers with no warm caches and restricted egress; corporate proxies env vars making urllib.request.urlopen loopback calls fail; a readiness route renamed between proxy versions; slow disk/network for tiktoken cache.

Related errors

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/a42c7f8c2fb16184. Report an issue: GitHub.