{"record":{"id":"3d94b7af8757c41b","repo":"nexu-io/open-design","slug":"reddit-rate-limited-429-fetching-url","errorCode":null,"errorMessage":"Reddit rate limited (429) fetching {url}","messagePattern":"Reddit rate limited \\(429\\) fetching (.+?)","errorType":"http","errorClass":"RedditRateLimitError","httpStatus":429,"severity":"error","filePath":"design-templates/last30days/scripts/lib/reddit_enrich.py","lineNumber":67,"sourceCode":"    Returns:\n        Thread data dict or None on failure\n\n    Raises:\n        RedditRateLimitError: When Reddit returns 429 (caller should bail)\n    \"\"\"\n    if mock_data is not None:\n        return mock_data\n\n    path = extract_reddit_path(url)\n    if not path:\n        return None\n\n    try:\n        data = http.get_reddit_json(path, timeout=timeout, retries=retries)\n        return data\n    except http.HTTPError as e:\n        if e.status_code == 429:\n            raise RedditRateLimitError(f\"Reddit rate limited (429) fetching {url}\") from e\n        return None\n\n\ndef parse_thread_data(data: Any) -> Dict[str, Any]:\n    \"\"\"Parse Reddit thread JSON into structured data.\n\n    Args:\n        data: Raw Reddit JSON response\n\n    Returns:\n        Dict with submission and comments data\n    \"\"\"\n    result = {\n        \"submission\": None,\n        \"comments\": [],\n    }\n\n    if not isinstance(data, list) or len(data) < 1:","sourceCodeStart":49,"sourceCodeEnd":85,"githubUrl":"https://github.com/nexu-io/open-design/blob/5be4028344c2eb4c667c5a97bda8f750c5597ef7/design-templates/last30days/scripts/lib/reddit_enrich.py#L49-L85","documentation":"RedditRateLimitError raised in fetch_thread_data when http.get_reddit_json raises an HTTPError with status_code 429. Unlike other HTTP errors (which return None and are silently tolerated), 429 is re-thrown so the caller can bail out of Reddit enrichment entirely rather than hammering a rate-limited endpoint.","triggerScenarios":"Calling fetch_thread_data on a reddit.com/.json URL when Reddit's edge returns 429 (Too Many Requests). This backend is the free fallback; the preferred ScrapeCreators backend does not rate-limit this way.","commonSituations":"Bulk-enriching many Reddit threads without the ScrapeCreators API key. Running from a shared IP that Reddit has throttled. Bursty concurrent calls exceeding Reddit's anonymous rate budget.","solutions":["Set SCRAPECREATORS_API_KEY so the pipeline prefers the rate-limit-free ScrapeCreators Reddit backend.","Catch RedditRateLimitError and back off (exponential sleep) or skip Reddit enrichment for this run.","Reduce concurrency / thread volume when using the free reddit.com/.json backend."],"exampleFix":"# before\ntry:\n    data = fetch_thread_data(url)\nexcept RedditRateLimitError:\n    pass  # swallowed, next call also 429s\n\n# after\nfrom lib.reddit_enrich import RedditRateLimitError\nimport time\ntry:\n    data = fetch_thread_data(url)\nexcept RedditRateLimitError:\n    time.sleep(60)\n    raise","handlingStrategy":"try-catch","validationCode":"null","typeGuard":"null","tryCatchPattern":"from lib.reddit_enrich import RedditRateLimitError, fetch_thread_data\n\ntry:\n    data = fetch_thread_data(url)\nexcept RedditRateLimitError:\n    # Reddit is throttled; stop enriching Reddit for this run.\n    data = None","preventionTips":["Prefer the ScrapeCreators Reddit backend (set SCRAPECREATORS_API_KEY) to avoid reddit.com 429s entirely.","Catch RedditRateLimitError specifically and back off or skip, do not let it abort the whole pipeline.","Throttle concurrency on the free reddit.com/.json backend."],"tags":["network","rate-limit","reddit","http"],"backgroundTag":null,"analyzedSha":"5be4028344c2eb4c667c5a97bda8f750c5597ef7","analyzedAt":"2026-08-12T12:03:58.812Z","schemaVersion":2},"datasetVersion":"2026-08-12T18:17:37.767Z"}