{"record":{"id":"e56db25806a98d0c","repo":"we-promise/sure","slug":"tiingo-hourly-request-limit-reached-new-count","errorCode":null,"errorMessage":"Tiingo hourly request limit reached (#{new_count}/#{max_requests_per_hour})","messagePattern":"Tiingo hourly request limit reached \\(#(.+?)/#(.+?)\\)","errorType":"exception","errorClass":"Provider::Tiingo::RateLimitError","httpStatus":null,"severity":"warning","filePath":"app/models/provider/tiingo.rb","lineNumber":245,"sourceCode":"\n        faraday.request :json\n        faraday.response :raise_error\n        faraday.headers[\"Authorization\"] = \"Token #{api_key}\"\n        faraday.headers[\"Content-Type\"] = \"application/json\"\n      end\n    end\n\n    # Adds hourly request counter on top of the interval throttle from RateLimitable.\n    def throttle_request\n      super\n\n      # Global per-hour request counter via cache (Redis).\n      # Atomic increment-then-check avoids the TOCTOU of read-check-increment.\n      hour_key = \"tiingo:requests:#{Time.current.to_i / 3600}\"\n      new_count = Rails.cache.increment(hour_key, 1, expires_in: 7200.seconds).to_i\n\n      if new_count >= max_requests_per_hour\n        raise RateLimitError, \"Tiingo hourly request limit reached (#{new_count}/#{max_requests_per_hour})\"\n      end\n    end\n\n    # Tracks unique symbols queried per month to stay within Tiingo's 500 symbols/month limit.\n    # Uses atomic set-if-absent (Redis SETNX) to eliminate the read-then-write race\n    # where two concurrent workers could both see the symbol as untracked and both\n    # increment the counter.\n    def track_symbol(symbol)\n      symbol_key = \"tiingo:symbol:#{Date.current.strftime('%Y-%m')}:#{symbol.upcase}\"\n      count_key  = \"tiingo:symbol_count:#{Date.current.strftime('%Y-%m')}\"\n\n      # Atomic write-if-absent: returns false when the key already exists (Redis SETNX).\n      # Only the first worker to claim this symbol will proceed to increment the counter.\n      return unless Rails.cache.write(symbol_key, true, expires_in: 35.days, unless_exist: true)\n\n      new_count = Rails.cache.increment(count_key, 1, expires_in: 35.days).to_i\n\n      if new_count >= MAX_SYMBOLS_PER_MONTH","sourceCodeStart":227,"sourceCodeEnd":263,"githubUrl":"https://github.com/we-promise/sure/blob/e69894adb92547273377398c15f45c979cd9416a/app/models/provider/tiingo.rb#L227-L263","documentation":"This is a client-side quota gate, not a server error. throttle_request first applies the interval throttle from RateLimitable, then atomically increments a Redis counter keyed 'tiingo:requests:<epoch-hour>' (expires in 7200s). When the returned count is >= max_requests_per_hour (ENV TIINGO_MAX_REQUESTS_PER_HOUR, default MAX_REQUESTS_PER_HOUR = 1000), it raises RateLimitError. The raise happens BEFORE the HTTP request, so no API call is wasted and Tiingo never saw the blocked request.","triggerScenarios":"Any Tiingo call (search_securities, fetch_security_prices, fetch_security_price, healthy?) once the whole app (all processes share the Redis counter) has made ~1000 Tiingo requests within the current wall-clock hour. The message includes the live count over the cap, e.g. '(1000/1000)'.","commonSituations":"Bulk price-refresh jobs sweeping large portfolios; a retry storm elsewhere re-requesting prices; default 1000/hr cap left in place while the account is on a paid tier that allows more; cache misses forcing repeated search calls.","solutions":["Wait for the hour window to roll over -- the key is 'tiingo:requests:#{Time.current.to_i / 3600}', so the counter resets at the next epoch-hour boundary","If your Tiingo plan allows more, raise TIINGO_MAX_REQUESTS_PER_HOUR in the environment","Reduce request volume: widen cache TTLs, batch date ranges into single fetch_security_prices calls instead of per-date fetch_security_price, reuse search currency caching"],"exampleFix":"# before (per-date lookups, one request each)\ndates.each { |d| provider.fetch_security_price(symbol: sym, date: d) }\n\n# after (one ranged request)\nprovider.fetch_security_prices(symbol: sym, start_date: dates.first, end_date: dates.last)","handlingStrategy":"retry","validationCode":"# Check the shared counter before issuing a request\ncount = Rails.cache.read(\"tiingo:requests:#{Time.current.to_i / 3600}\").to_i\nraise Provider::Tiingo::RateLimitError, 'Local hourly budget spent' if count >= ENV.fetch('TIINGO_MAX_REQUESTS_PER_HOUR', 1000).to_i","typeGuard":null,"tryCatchPattern":"begin\n  provider.fetch_security_prices(...)\nrescue Provider::Tiingo::RateLimitError\n  SyncRetryJob.set(wait: until_next_epoch_hour).perform_later(...) # window resets on the hour\nend","preventionTips":["Batch date ranges into single ranged calls instead of per-date requests","Set TIINGO_MAX_REQUESTS_PER_HOUR to the real plan limit, not the default","Prefer cached currency/search data to avoid repeat search calls"],"tags":["tiingo","rate-limit","client-side-throttle","redis-counter","quota"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"e69894adb92547273377398c15f45c979cd9416a","analyzedAt":"2026-08-21T18:22:41.165Z","schemaVersion":2},"datasetVersion":"2026-08-21T23:17:16.201Z"}