{"record":{"id":"dc760f1d89f2af65","repo":"we-promise/sure","slug":"yahoo-finance-rate-limit-exceeded","errorCode":null,"errorMessage":"Yahoo Finance rate limit exceeded","messagePattern":"Yahoo Finance rate limit exceeded","errorType":"exception","errorClass":"Provider::YahooFinance::RateLimitError","httpStatus":429,"severity":"warning","filePath":"app/models/provider/yahoo_finance.rb","lineNumber":879,"sourceCode":"    # ================================\n\n    # Fetches and caches the Yahoo Finance cookie and crumb for authenticated endpoints\n    # The crumb is a CSRF token required by some Yahoo Finance endpoints (e.g., quoteSummary)\n    def fetch_cookie_and_crumb\n      cache_key = \"#{@cache_prefix}_auth_crumb\"\n      cached = Rails.cache.read(cache_key)\n      if cached.present?\n        return cached if valid_crumb?(cached.second)\n\n        Rails.cache.delete(cache_key)\n      end\n\n      cookie, crumb, cache_duration = request_cookie_and_crumb(auth_client)\n      result = [ cookie, crumb ]\n      Rails.cache.write(cache_key, result, expires_in: cache_duration)\n      result\n    rescue Faraday::TooManyRequestsError => e\n      raise RateLimitError.new(\n        \"Yahoo Finance rate limit exceeded\",\n        details: { status: e.response&.dig(:status) }\n      )\n    rescue Faraday::Error => e\n      raise AuthenticationError, \"Failed to authenticate with Yahoo Finance: #{e.message}\"\n    end\n\n    def request_cookie_and_crumb(authentication_client)\n      cookie_response = authentication_client.get(\"https://fc.yahoo.com\")\n      if cookie_response.respond_to?(:status) && cookie_response.status == 429\n        raise RateLimitError.new(\n          \"Yahoo Finance rate limit exceeded\",\n          details: { status: cookie_response.status }\n        )\n      end\n\n      cookie = extract_cookie(cookie_response)\n      cookie_max_age = extract_cookie_max_age(cookie_response)","sourceCodeStart":861,"sourceCodeEnd":897,"githubUrl":"https://github.com/we-promise/sure/blob/e69894adb92547273377398c15f45c979cd9416a/app/models/provider/yahoo_finance.rb#L861-L897","documentation":"During Yahoo Finance cookie+crumb bootstrapping, a Faraday::TooManyRequestsError (HTTP 429 from any Faraday request in authenticate) is re-raised as Provider::YahooFinance::RateLimitError with the status in details. Yahoo rate-limits unauthenticated scraping traffic; this error means the auth handshake itself was throttled before any quote data was fetched.","triggerScenarios":"The cached [cookie, crumb] pair expired (or was invalidated by valid_crumb?) so request_cookie_and_crumb re-runs, and a Faraday GET (fc.yahoo.com or query1.finance.yahoo.com/v1/test/getcrumb) returns 429. Common when syncing many securities in tight loops from one IP.","commonSituations":"Scheduled price-sync jobs for large watchlists hammering Yahoo without spacing; running the app from a shared/cloud IP (AWS, CI runner) that Yahoo already throttles; the cache entry evicted early (Rails.cache full) causing repeated handshakes; stale crumb forcing re-auth on every request.","solutions":["Honor MIN_REQUEST_INTERVAL (>= 0.5s between Yahoo requests) and back off on RateLimitError for several minutes","Check Rails.cache is functioning - a broken cache means every call re-does the cookie/crumb handshake","Reduce the security universe per cycle or stagger sync jobs with jitter","If on a cloud/shared egress IP, route Yahoo traffic through a residential/stable IP or accept longer backoff windows"],"exampleFix":"# before\n10.times { Provider::YahooFinance.new.fetch_quote(symbol) }\n\n# after\nyh = Provider::YahooFinance.new\n10.times do |i|\n  sleep(Provider::YahooFinance::MIN_REQUEST_INTERVAL)\n  begin\n    yh.fetch_quote(symbol)\n  rescue Provider::YahooFinance::RateLimitError\n    sleep(300) && retry if (tries += 1) < 2\n    raise\n  end\nend","handlingStrategy":"retry","validationCode":"null","typeGuard":"def yahoo_rate_limited?(err)\n  err.is_a?(Provider::YahooFinance::RateLimitError)\nend","tryCatchPattern":"retries = 0\nbegin\n  provider.fetch_exchange_rate(from, to, date)\nrescue Provider::YahooFinance::RateLimitError\n  raise if (retries += 1) > 2\n  sleep(5.minutes)\n  retry\nend","preventionTips":["Space Yahoo calls at least MIN_REQUEST_INTERVAL (0.5s) apart","Cache cookie/crumb (already 1h TTL) and quotes (5m TTL) - never handshake per request","On shared infrastructure, dedupe Yahoo work behind one queue worker"],"tags":["yahoo-finance","rate-limit","http-429","scraping","cookie-crumb"],"backgroundTag":"http-429-rate-limit","analyzedSha":"e69894adb92547273377398c15f45c979cd9416a","analyzedAt":"2026-08-21T18:22:41.165Z","schemaVersion":2},"datasetVersion":"2026-08-21T23:17:16.201Z"}