{"record":{"id":"b13a4a6398c787e9","repo":"Z4nzu/hackingtool","slug":"github-search-rate-limit-reached-wait","errorCode":null,"errorMessage":"GitHub search rate limit reached{wait}","messagePattern":"GitHub search rate limit reached(.+?)","errorType":"exception","errorClass":"RateLimited","httpStatus":null,"severity":"warning","filePath":"src/hackingtool/discover.py","lineNumber":523,"sourceCode":"        \"Accept\": \"application/vnd.github+json\",\n        \"X-GitHub-Api-Version\": \"2022-11-28\",\n        \"User-Agent\": _USER_AGENT,\n    })\n    tok = _token()\n    if tok:\n        req.add_header(\"Authorization\", f\"Bearer {tok}\")\n    try:\n        with urllib.request.urlopen(req, timeout=_TIMEOUT) as resp:\n            return json.loads(resp.read().decode(\"utf-8\"))\n    except urllib.error.HTTPError as exc:\n        if exc.code in (403, 429) and exc.headers.get(\"x-ratelimit-remaining\") == \"0\":\n            reset = exc.headers.get(\"x-ratelimit-reset\", \"\")\n            wait = \"\"\n            try:\n                wait = f\" (~{max(0, int(reset) - int(time.time()))}s)\"\n            except ValueError:\n                pass\n            raise RateLimited(f\"GitHub search rate limit reached{wait}\") from exc\n        raise\n\n\ndef _to_repo(item: dict) -> Repo:\n    \"\"\"Read ONLY the allowlisted fields. Nothing else is touched.\"\"\"\n    lic = item.get(\"license\") or {}\n    owner = item.get(\"owner\") or {}\n    return Repo(\n        full_name=item.get(\"full_name\", \"\"),\n        description=(item.get(\"description\") or \"\").strip(),\n        url=item.get(\"html_url\", \"\"),\n        stars=int(item.get(\"stargazers_count\") or 0),\n        forks=int(item.get(\"forks_count\") or 0),\n        pushed_at=item.get(\"pushed_at\") or \"\",\n        created_at=item.get(\"created_at\") or \"\",\n        archived=bool(item.get(\"archived\")) or bool(item.get(\"disabled\")),\n        fork=bool(item.get(\"fork\")),\n        license=(lic.get(\"spdx_id\") or \"\") if lic.get(\"spdx_id\") != \"NOASSERTION\" else \"\",","sourceCodeStart":505,"sourceCodeEnd":541,"githubUrl":"https://github.com/Z4nzu/hackingtool/blob/9b92b6156ddc5ff87bf0fa592d81c333a88d1c78/src/hackingtool/discover.py#L505-L541","documentation":"RateLimited is raised by hackingtool's GitHub repository search (_fetch in discover.py:502) when the GitHub Search API responds with HTTP 403 or 429 and the response header x-ratelimit-remaining is exactly \"0\" — i.e. the caller has exhausted GitHub's primary rate limit for the search endpoint. The library deliberately rethrows this as a distinct exception (instead of returning [] like other failures) so callers can tell 'no results / transient error' apart from 'you must wait'. The message embeds a computed wait hint derived from the x-ratelimit-reset header, e.g. \"GitHub search rate limit reached (~58s)\".","triggerScenarios":"Any call chain that reaches discover._search(query) with a cache miss (the 24h on-disk cache in USER_CONFIG_FILE.parent/cache/find/ is empty or expired for that query): it builds a request to the GitHub search/repositories endpoint with per_page=10, and GitHub replies 403 (unauthenticated) or 429 with x-ratelimit-remaining: 0. Without a token the search limit is only 10 requests/minute, so ~10 distinct queries in a minute trigger it; with a token it is 30/minute. The 403+remaining=0 combination is how GitHub reports exhausted unauthenticated quotas (429 is the authenticated form), which is why both codes are checked in discover.py:516.","commonSituations":"Running hackingtool find repeatedly with different phrasings in a short session while unauthenticated (most common). CI or shared NAT/proxy egress where the 60-requests/hour anonymous pool is consumed by others on the same IP. Providing a GitHub token that is expired, revoked, or lacks access, causing GitHub to treat requests as anonymous. Clock skew or an old cached process holding a stale token. Multi-user scripts or loops that call the search CLI per-item without backoff.","solutions":["Set a GitHub token: export HACKINGTOOL_GITHUB_TOKEN (or GITHUB_TOKEN / GH_TOKEN) — this raises the search limit from 10 to 30 requests/minute; create one at https://github.com/settings/tokens (no scopes needed for public search).","Wait out the window shown in the message (the \"(~Ns)\" suffix is seconds until x-ratelimit-reset) and retry the same query — the 24h on-disk cache means a successful retry costs nothing on repeat.","Reduce distinct queries: re-run the same wording instead of new phrasings, since cache hits never touch the network.","If the token is set but the error persists, verify it is valid (gh auth status or curl -H \"Authorization: Bearer $TOKEN\" https://api.github.com/rate_limit) — an invalid token silently downgrades you to the anonymous quota.","In scripts/loops, wrap calls in a RateLimited handler that sleeps for the reported seconds and retries once (see tryCatchPattern)."],"exampleFix":"# before: unauthenticated, 10 search req/min\n$ hackingtool find \"xss scanner\"\n$ hackingtool find \"sql injection scanner\"\n$ hackingtool find \"subdomain scanner\"   # -> RateLimited: GitHub search rate limit reached (~52s)\n\n# after: authenticated, 30 req/min\n$ export HACKINGTOOL_GITHUB_TOKEN=ghp_your_personal_access_token\n$ hackingtool find \"xss scanner\"","handlingStrategy":"retry","validationCode":"import json, os, time, urllib.request\n\ndef github_search_quota_left() -> int | None:\n    \"\"\"Check https://api.github.com/rate_limit before searching. None = unknown.\"\"\"\n    tok = (os.environ.get(\"HACKINGTOOL_GITHUB_TOKEN\")\n           or os.environ.get(\"GITHUB_TOKEN\")\n           or os.environ.get(\"GH_TOKEN\") or \"\").strip()\n    req = urllib.request.Request(\"https://api.github.com/rate_limit\")\n    if tok:\n        req.add_header(\"Authorization\", f\"Bearer {tok}\")\n    try:\n        with urllib.request.urlopen(req, timeout=10) as r:\n            core = json.load(r)[\"resources\"][\"search\"]\n            return core[\"remaining\"]\n    except Exception:\n        return None\n\nif github_search_quota_left() == 0:\n    raise SystemExit(\"search quota exhausted; wait or set HACKINGTOOL_GITHUB_TOKEN\")\nfrom hackingtool import discover\ndiscover._search(\"topic:xss\")","typeGuard":"from hackingtool import discover\n\ndef is_rate_limited(exc: BaseException) -> bool:\n    \"\"\"True if exc is hackingtool's GitHub search rate-limit error.\"\"\"\n    return isinstance(exc, discover.RateLimited)","tryCatchPattern":"import re, time\nfrom hackingtool import discover\n\n_WAIT_RX = re.compile(r\"~(\\d+)s\")\n\ntry:\n    repos = discover._search(\"topic:security language:python\")\nexcept discover.RateLimited as exc:\n    m = _WAIT_RX.search(str(exc))\n    delay = int(m.group(1)) + 2 if m else 60          # header-derived wait, safe floor\n    time.sleep(delay)\n    repos = discover._search(\"topic:security language:python\")  # cache makes retry cheap\nexcept Exception:\n    repos = []                                         # other failures degrade gracefully\n# proceed with repos","preventionTips":["Always export HACKINGTOOL_GITHUB_TOKEN (a fine-grained token with no scopes works for public search) in shells, CI env, and systemd units that run hackingtool find.","Batch exploratory queries instead of firing many phrasings back-to-back; unauthenticated search allows only 10 requests/minute.","Rely on the built-in 24h per-query cache: repeat identical wording rather than generating new variants of the same query.","Check https://api.github.com/rate_limit (search bucket) before long batch jobs and abort early with a clear message instead of mid-run.","Catch RateLimited specifically and sleep for the reported \"~Ns\" wait hint plus a small buffer, then retry once — never blanket-retry on generic HTTPError, since that hides real failures.","In multi-worker setups, share one token and a shared cache directory so workers don't each burn the anonymous IP quota."],"tags":["rate-limit","github-api","network","http-403","http-429","retry"],"backgroundTag":null,"analyzedSha":"9b92b6156ddc5ff87bf0fa592d81c333a88d1c78","analyzedAt":"2026-08-14T20:41:12.589Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}