{"record":{"id":"388a81bd46a2b4a0","repo":"headroomlabs-ai/headroom","slug":"failed-to-download-final-url-after-attempts-at","errorCode":null,"errorMessage":"failed to download {final_url} after {attempts} attempts: {e}","messagePattern":"failed to download (.+?) after (.+?) attempts: (.+?)","errorType":"exception","errorClass":"BinaryFetchError","httpStatus":null,"severity":"error","filePath":"headroom/binaries.py","lineNumber":259,"sourceCode":"        raise OfflineError(f\"offline mode (HEADROOM_BINARIES_OFFLINE=1) but fetch required: {url}\")\n    if not _has_writable_existing_parent(dest.parent):\n        raise OSError(f\"binary cache directory parent is not writable: {dest.parent}\")\n    dest.parent.mkdir(parents=True, exist_ok=True)\n    if not _is_writable_dir(dest.parent):\n        raise OSError(f\"binary cache directory is not writable: {dest.parent}\")\n    final_url = _mirror_url(url)\n    req = urllib.request.Request(final_url, headers={\"User-Agent\": \"headroom-binaries/1\"})\n    attempts = 3\n    for attempt in range(1, attempts + 1):\n        try:\n            with urllib.request.urlopen(req, timeout=60) as resp:  # noqa: S310 (https)\n                total = int(resp.headers.get(\"Content-Length\") or 0)\n                _stream_to(resp, dest, total, label=dest.name, show_progress=progress)\n            return\n        except urllib.error.URLError as e:\n            dest.unlink(missing_ok=True)\n            if attempt == attempts:\n                raise BinaryFetchError(\n                    f\"failed to download {final_url} after {attempts} attempts: {e}\"\n                ) from e\n            # GitHub release assets occasionally return a transient 5xx or\n            # reset while redirecting to the object store. A short bounded\n            # retry keeps proxy startup reliable without hiding persistent\n            # credential, mirror, or connectivity failures.\n            time.sleep(0.25 * attempt)\n\n\ndef _stream_to(src: Any, dest: Path, total: int, *, label: str, show_progress: bool) -> None:\n    # Rich progress if available and stderr is a tty; otherwise silent chunked copy.\n    try:\n        if show_progress and sys.stderr.isatty():\n            from rich.progress import (\n                BarColumn,\n                DownloadColumn,\n                Progress,\n                TextColumn,","sourceCodeStart":241,"sourceCodeEnd":277,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/binaries.py#L241-L277","documentation":"_download retries each urllib.error.URLError up to 3 times (linear backoff 0.25s * attempt) because GitHub release assets intermittently 5xx or reset during the redirect to the object store. If the final attempt still fails, BinaryFetchError is raised with the (mirror-substituted) final URL and the underlying exception chained via 'from e'. Persistent failures — DNS, proxy, auth, mirror misconfig — are never masked by the retry.","triggerScenarios":"Any URLError persisting across 3 attempts: no network egress, corporate proxy blocking github.com/objects.githubusercontent.com, HEADROOM_BINARIES_MIRROR pointing at a dead endpoint, or TLS interception failures.","commonSituations":"CI runners without GitHub egress, air-gapped clusters with an incorrectly configured internal mirror URL, corporate SSL-inspection proxies breaking the asset redirect chain.","solutions":["Read the chained '{e}' — 'Name or service not known' means DNS, '407/403' means proxy/auth, 'CERTIFICATE_VERIFY_FAILED' means TLS interception.","If direct GitHub is blocked, set HEADROOM_BINARIES_MIRROR to an internal mirror that serves the same path layout (only github.com and objects.githubusercontent.com prefixes are rewritten).","Allow egress to github.com and objects.githubusercontent.com in firewall/proxy rules.","If TLS interception is the cause, install the corporate CA into the image trust store."],"exampleFix":"# before\nENV HEADROOM_BINARIES_MIRROR=https://artifacts.internal/gh  # 404s -> BinaryFetchError\n\n# after\n# mirror must serve the same path layout as github.com releases:\nENV HEADROOM_BINARIES_MIRROR=https://artifacts.internal/github-mirror","handlingStrategy":"retry","validationCode":"import socket, urllib.request\n\ndef github_reachable() -> bool:\n    for host in (\"github.com\", \"objects.githubusercontent.com\"):\n        try:\n            urllib.request.urlopen(f\"https://{host}\", timeout=5)\n        except urllib.error.URLError:\n            return False\n    return True\n\nif not github_reachable() and not os.environ.get(\"HEADROOM_BINARIES_MIRROR\"):\n    raise SystemExit(\"no egress to GitHub; set HEADROOM_BINARIES_MIRROR to an internal mirror\")","typeGuard":null,"tryCatchPattern":"from headroom.binaries import BinaryFetchError\n\nfor attempt in range(2):\n    try:\n        ensure_binary(tool)\n        break\n    except BinaryFetchError as e:\n        if attempt == 1:\n            raise SystemExit(f\"persistent fetch failure: {e}; check proxy/mirror/egress\") from e\n        time.sleep(5)  # outer retry only for transient infra blips","preventionTips":["Open egress to github.com and objects.githubusercontent.com (or configure a mirror with identical paths).","The internal retry is only 3 quick attempts; do not rely on it for flaky networks — add health monitoring.","Verify corporate proxies allow the release-asset redirect chain, not just the initial request."],"tags":["python","binaries","network","download","retry","proxy","mirror"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}