fluent/fluentd · critical · RetryLimitError

exceed retry limit

Error message

exceed retry limit

What it means

Raised as RetryLimitError by Writer#try_connect in fluent-cat. Every failed connect attempt to the forward socket (TCP 127.0.0.1:24224 by default, or a unix socket with -u) is pushed onto @error_history; once its size exceeds --retry-limit (default 5), all pending records are dumped to stdout as '!<time>:<json>' lines via abort_message, the buffers are cleared, and 'exceed retry limit' propagates, terminating fluent-cat with the error printed.

Source

Thrown at lib/fluent/command/cat.rb:299

      @socket = @connector.call
      @error_history.clear
      return true

    rescue RetryLimitError => ex
      raise ex
    rescue
      $stderr.puts "connect failed: #{$!}"
      @error_history << $!
      @socket_time = now

      if @retry_limit < @error_history.size
        # abort all pending records
        @pending.each {|(time, record)|
          abort_message(time, record)
        }
        @pending.clear
        @error_history.clear
        raise RetryLimitError, "exceed retry limit"
      else
        retry
      end
    end
  end

  def abort_message(time, record)
    $stdout.puts "!#{time}:#{JSON.generate(record)}"
  end
end


if unix
  connector = Proc.new {
    UNIXSocket.open(socket_path)
  }
else
  connector = Proc.new {

View on GitHub (pinned to dd45c6e18d)

Solutions

1. Verify the endpoint before piping: nc -vz <host> 24224 (or check the unix socket file exists)
2. Start fluentd / enable the in_forward plugin on the expected host, port and bind
3. Raise tolerance for short outages: fluent-cat --retry-limit N (pending records beyond 1024 are still dropped with '!'-prefixed stdout lines)
4. Recover aborted records from the '!<time>:<json>' stdout lines and resend them after the endpoint is back

Example fix

# before
cat events.jsonl | fluent-cat my.tag   # fluentd down -> RetryLimitError after 5 tries

# after
nc -vz 127.0.0.1 24224                 # verify in_forward first
cat events.jsonl | fluent-cat --retry-limit 30 my.tag

# gate the pipe on endpoint reachability, then send with more retries
system('nc -z 127.0.0.1 24224') or abort 'fluentd forward socket not reachable'
# cat events.jsonl | fluent-cat --retry-limit 30 my.tag

When it happens

Trigger: fluentd is not running or in_forward is disabled; wrong -h/-p values; firewall or bind-address restrictions on 24224; wrong unix socket path with -u/-s; the remote forwarder stays unreachable longer than the retry window while records queue up (pending cap 1024).

Common situations: Shipping logs from a container/host where fluentd crashed; port 24224 exposed only on another interface; SELinux/firewalld blocking the socket; startup-order races where fluent-cat runs before fluentd.

Related errors

• Input must be a map (got #{record.class})
• failed to flush the buffer with secondary output.
• invalid_params
• unknown_key
• Unknown #{@kind} plugin '#{type}'. Run 'gem search -rd fluen

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/8b3f5a5561ffa3ce. Report an issue: GitHub.