we-promise/sure · warning · Provider::YahooFinance::RateLimitError

Yahoo Finance rate limit exceeded

Error message

Yahoo Finance rate limit exceeded

What it means

During Yahoo Finance cookie+crumb bootstrapping, a Faraday::TooManyRequestsError (HTTP 429 from any Faraday request in authenticate) is re-raised as Provider::YahooFinance::RateLimitError with the status in details. Yahoo rate-limits unauthenticated scraping traffic; this error means the auth handshake itself was throttled before any quote data was fetched.

Source

Thrown at app/models/provider/yahoo_finance.rb:879

    # ================================

    # Fetches and caches the Yahoo Finance cookie and crumb for authenticated endpoints
    # The crumb is a CSRF token required by some Yahoo Finance endpoints (e.g., quoteSummary)
    def fetch_cookie_and_crumb
      cache_key = "#{@cache_prefix}_auth_crumb"
      cached = Rails.cache.read(cache_key)
      if cached.present?
        return cached if valid_crumb?(cached.second)

        Rails.cache.delete(cache_key)
      end

      cookie, crumb, cache_duration = request_cookie_and_crumb(auth_client)
      result = [ cookie, crumb ]
      Rails.cache.write(cache_key, result, expires_in: cache_duration)
      result
    rescue Faraday::TooManyRequestsError => e
      raise RateLimitError.new(
        "Yahoo Finance rate limit exceeded",
        details: { status: e.response&.dig(:status) }
      )
    rescue Faraday::Error => e
      raise AuthenticationError, "Failed to authenticate with Yahoo Finance: #{e.message}"
    end

    def request_cookie_and_crumb(authentication_client)
      cookie_response = authentication_client.get("https://fc.yahoo.com")
      if cookie_response.respond_to?(:status) && cookie_response.status == 429
        raise RateLimitError.new(
          "Yahoo Finance rate limit exceeded",
          details: { status: cookie_response.status }
        )
      end

      cookie = extract_cookie(cookie_response)
      cookie_max_age = extract_cookie_max_age(cookie_response)

View on GitHub (pinned to e69894adb9)

Solutions

1. Honor MIN_REQUEST_INTERVAL (>= 0.5s between Yahoo requests) and back off on RateLimitError for several minutes
2. Check Rails.cache is functioning - a broken cache means every call re-does the cookie/crumb handshake
3. Reduce the security universe per cycle or stagger sync jobs with jitter
4. If on a cloud/shared egress IP, route Yahoo traffic through a residential/stable IP or accept longer backoff windows

Example fix

# before
10.times { Provider::YahooFinance.new.fetch_quote(symbol) }

# after
yh = Provider::YahooFinance.new
10.times do |i|
  sleep(Provider::YahooFinance::MIN_REQUEST_INTERVAL)
  begin
    yh.fetch_quote(symbol)
  rescue Provider::YahooFinance::RateLimitError
    sleep(300) && retry if (tries += 1) < 2
    raise
  end
end

null

def yahoo_rate_limited?(err)
  err.is_a?(Provider::YahooFinance::RateLimitError)
end

retries = 0
begin
  provider.fetch_exchange_rate(from, to, date)
rescue Provider::YahooFinance::RateLimitError
  raise if (retries += 1) > 2
  sleep(5.minutes)
  retry
end

When it happens

Trigger: The cached [cookie, crumb] pair expired (or was invalidated by valid_crumb?) so request_cookie_and_crumb re-runs, and a Faraday GET (fc.yahoo.com or query1.finance.yahoo.com/v1/test/getcrumb) returns 429. Common when syncing many securities in tight loops from one IP.

Common situations: Scheduled price-sync jobs for large watchlists hammering Yahoo without spacing; running the app from a shared/cloud IP (AWS, CI runner) that Yahoo already throttles; the cache entry evicted early (Rails.cache full) causing repeated handshakes; stale crumb forcing re-auth on every request.

Related errors

• Could not save that passkey or security key. Please try agai
• invalid_import_record
• rate_limited
• rate_limited
• rate_limited

AI-assisted analysis of we-promise/sure@e69894adb9 (2026-08-21). Data as JSON: /api/errors/dc760f1d89f2af65. Report an issue: GitHub.