{"record":{"id":"3b9286a8c0519a5f","repo":"unslothai/unsloth","slug":"rest-failed-after-max-retries-retries-last-err","errorCode":null,"errorMessage":"REST failed after {max_retries} retries: {last_err}","messagePattern":"REST failed after (.+?) retries: (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"studio/backend/plugins/data-designer-github-repo-seed/src/data_designer_github_repo_seed/scraper_impl/gh_client.py","lineNumber":280,"sourceCode":"                    retry_after = _retry_after_seconds(r.headers.get(\"Retry-After\"))\n                    if retry_after is not None:\n                        log.warning(\"Secondary rate limit on REST. Sleep %ds.\", retry_after)\n                        time.sleep(retry_after + 2)\n                        continue\n                    # Primary rate limit\n                    if self.rest_remaining == 0 and self.rest_reset:\n                        self._sleep_until(self.rest_reset)\n                        continue\n                    log.warning(\"REST 403/429, sleep 60\")\n                    time.sleep(60)\n                    continue\n                return r\n            except requests.RequestException as e:\n                last_err = e\n                log.warning(\"REST network error: %s. Retry.\", e)\n                time.sleep(backoff)\n                backoff = min(backoff * 2, 60)\n        raise RuntimeError(f\"REST failed after {max_retries} retries: {last_err}\")\n\n    def rest_paginate(\n        self,\n        path: str,\n        params: Optional[Dict[str, Any]] = None,\n        per_page: int = 100,\n    ) -> Iterator[dict]:\n        params = dict(params or {})\n        params.setdefault(\"per_page\", per_page)\n        url = path\n        while True:\n            r = self.rest(\"GET\", url, params = params if url == path else None)\n            if r.status_code != 200:\n                log.error(\"REST paginate got %s at %s: %s\", r.status_code, url, r.text[:200])\n                return\n            items = r.json()\n            if isinstance(items, dict):\n                # Some endpoints wrap the list in an \"items\" field","sourceCodeStart":262,"sourceCodeEnd":298,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/plugins/data-designer-github-repo-seed/src/data_designer_github_repo_seed/scraper_impl/gh_client.py#L262-L298","documentation":"RuntimeError raised after the REST request exhausts max_retries attempts (default 6). Retries happen for transport-level requests.RequestException with exponential backoff (capped at 60s), and for 403/429 responses the client sleeps (until rate-limit reset if known, else 60s) before retrying. Only when every attempt fails does this error surface with the last underlying exception.","triggerScenarios":"Sustained rate limiting that outlasts the retry budget (repeated 403/429), or persistent network failures (connection refused, DNS, proxy) on every REST attempt for a paginated listing call.","commonSituations":"Scraping very large repos beyond the rate-limit budget (unauthenticated-range limits despite a token; secondary/abuse rate limits); running many parallel scraper instances sharing one token; network egress blocked in CI.","solutions":["Wait for the rate-limit window to reset (check X-RateLimit-Reset) and re-run; the scraper resumes from cached data.","Reduce parallelism or run a single scraper instance per token to stay under primary and secondary rate limits.","Fix network/proxy issues if the underlying exception is a connection error rather than 403/429.","Conditionally increase max_retries or use a token with a higher rate-limit tier (GitHub Apps)."],"exampleFix":"# before\n# 3 parallel scrapers, same GH_TOKEN -> REST failed after 6 retries: 403\n\n# after\n# single scraper instance; rerun after rate-limit reset\n$ python scrape.py --resume","handlingStrategy":"retry","validationCode":"def rate_budget_ok(remaining: int | None, reset_epoch: int | None, needed: int = 1) -> bool:\n    return remaining is None or remaining >= needed or bool(reset_epoch)","typeGuard":null,"tryCatchPattern":"try:\n    resp = client.rest(\"GET\", \"/repos/o/r/issues\")\nexcept RuntimeError as e:\n    if \"REST failed after\" in str(e) and \"403\" in str(e.__cause__ or \"\"):\n        wait_for_rate_limit_reset(client.rest_reset)  # sleep until reset epoch, rerun\n    else:\n        raise","preventionTips":["Monitor X-RateLimit-Remaining/Reset on responses and pause scraping near zero.","Run one scraper instance per token; serialize large repo harvests.","Make jobs resumable so a mid-run rate-limit exhaustion costs nothing on rerun."],"tags":["github","rest","rate-limit","network","retry"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}