puppetlabs/puppet · error · Puppet::HTTP::TooManyRetryAfters

Too many HTTP retries for %{addr}

Error message

Too many HTTP retries for %{addr}

What it means

When a server answers 429 or 503 with a Retry-After header, Puppet::HTTP::RetryAfterHandler sleeps for the indicated time (capped at Puppet[:runinterval]) and the client retries. Each attempt increments the count; when it reaches the client's retry_limit (default 100, set at Puppet::HTTP::Client.new), Puppet::HTTP::TooManyRetryAfters is raised with the request URI — the server never became healthy.

Source

Thrown at lib/puppet/http/retry_after_handler.rb:51

    else
      false
    end
  end

  # The amount of time to wait before attempting a retry
  #
  # @param [Net::HTTP] request
  # @param [Puppet::HTTP::Response] response
  # @param [Integer] retries number of retries attempted so far
  #
  # @return [Integer] the amount of time to wait
  #
  # @raise [Puppet::HTTP::TooManyRetryAfters] raise if we have hit our retry
  #   limit
  #
  # @api private
  def retry_after_interval(request, response, retries)
    raise Puppet::HTTP::TooManyRetryAfters, request.uri if retries >= @retry_limit

    retry_after = response['Retry-After']
    return nil unless retry_after

    seconds = parse_retry_after(retry_after)

    # if retry-after is far in the future, we could end up sleeping repeatedly
    # for 30 minutes, effectively waiting indefinitely, seems like we should wait
    # in total for 30 minutes, in which case this upper limit needs to be enforced
    # by the client.
    [seconds, @max_sleep].min
  end

  private

  def parse_retry_after(retry_after)
    Integer(retry_after)
  rescue TypeError, ArgumentError

View on GitHub (pinned to e227c27540)

Solutions

1. Check server health: the endpoint returning 503/429 is the root cause — resolve capacity or finish the maintenance window.
2. Inspect the Retry-After the server sends; if it is huge, waiting server-side beats burning the client retry budget.
3. If a transient storm is expected, construct Puppet::HTTP::Client.new(retry_limit: N) with a larger budget in that specific tool (not agents).
4. For rate limiting, stagger agent runs or spread load across server replicas.

Example fix

# before
client = Puppet::HTTP::Client.new

# after - tolerate a longer maintenance window in a maintenance script
client = Puppet::HTTP::Client.new(retry_limit: 200)

begin
  client.get(uri)
rescue Puppet::HTTP::TooManyRetryAfters => e
  sleep 300
  raise if (attempts += 1) > 3
  retry
end

When it happens

Trigger: A puppetserver/CA/LB persistently returning 503 (maintenance, overload, draining) or 429 rate-limiting across more than retry_limit consecutive retry cycles during one logical request.

Common situations: Puppet server restarting or upgrading while agents compile; rate limiters (Cloudflare, API gateways) throttling the agent; LB health checks flapping so backends keep toggling; runinterval short enough that the sleep cap makes retries cheap but the outage outlasts the budget.

Related errors

• Failed to parse Retry-After header '%{retry_after}' as an in
• 'put' requires a string 'body' argument
• 'put' requires a 'content-type' header
• 'post' requires a string 'body' argument
• 'post' requires a 'content-type' header

AI-assisted analysis of puppetlabs/puppet@e227c27540 (2026-08-21). Data as JSON: /api/errors/3cd6b80ee483ff15. Report an issue: GitHub.