{"record":{"id":"8b3f5a5561ffa3ce","repo":"fluent/fluentd","slug":"exceed-retry-limit","errorCode":null,"errorMessage":"exceed retry limit","messagePattern":"exceed retry limit","errorType":"exception","errorClass":"RetryLimitError","httpStatus":null,"severity":"critical","filePath":"lib/fluent/command/cat.rb","lineNumber":299,"sourceCode":"      @socket = @connector.call\n      @error_history.clear\n      return true\n\n    rescue RetryLimitError => ex\n      raise ex\n    rescue\n      $stderr.puts \"connect failed: #{$!}\"\n      @error_history << $!\n      @socket_time = now\n\n      if @retry_limit < @error_history.size\n        # abort all pending records\n        @pending.each {|(time, record)|\n          abort_message(time, record)\n        }\n        @pending.clear\n        @error_history.clear\n        raise RetryLimitError, \"exceed retry limit\"\n      else\n        retry\n      end\n    end\n  end\n\n  def abort_message(time, record)\n    $stdout.puts \"!#{time}:#{JSON.generate(record)}\"\n  end\nend\n\n\nif unix\n  connector = Proc.new {\n    UNIXSocket.open(socket_path)\n  }\nelse\n  connector = Proc.new {","sourceCodeStart":281,"sourceCodeEnd":317,"githubUrl":"https://github.com/fluent/fluentd/blob/dd45c6e18dc7be33b5e5a0f0767bf46307ff5626/lib/fluent/command/cat.rb#L281-L317","documentation":"Raised as RetryLimitError by Writer#try_connect in fluent-cat. Every failed connect attempt to the forward socket (TCP 127.0.0.1:24224 by default, or a unix socket with -u) is pushed onto @error_history; once its size exceeds --retry-limit (default 5), all pending records are dumped to stdout as '!<time>:<json>' lines via abort_message, the buffers are cleared, and 'exceed retry limit' propagates, terminating fluent-cat with the error printed.","triggerScenarios":"fluentd is not running or in_forward is disabled; wrong -h/-p values; firewall or bind-address restrictions on 24224; wrong unix socket path with -u/-s; the remote forwarder stays unreachable longer than the retry window while records queue up (pending cap 1024).","commonSituations":"Shipping logs from a container/host where fluentd crashed; port 24224 exposed only on another interface; SELinux/firewalld blocking the socket; startup-order races where fluent-cat runs before fluentd.","solutions":["Verify the endpoint before piping: nc -vz <host> 24224 (or check the unix socket file exists)","Start fluentd / enable the in_forward plugin on the expected host, port and bind","Raise tolerance for short outages: fluent-cat --retry-limit N (pending records beyond 1024 are still dropped with '!'-prefixed stdout lines)","Recover aborted records from the '!<time>:<json>' stdout lines and resend them after the endpoint is back"],"exampleFix":"# before\ncat events.jsonl | fluent-cat my.tag   # fluentd down -> RetryLimitError after 5 tries\n\n# after\nnc -vz 127.0.0.1 24224                 # verify in_forward first\ncat events.jsonl | fluent-cat --retry-limit 30 my.tag","handlingStrategy":"retry","validationCode":"# gate the pipe on endpoint reachability, then send with more retries\nsystem('nc -z 127.0.0.1 24224') or abort 'fluentd forward socket not reachable'\n# cat events.jsonl | fluent-cat --retry-limit 30 my.tag","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Check the socket (nc -vz host 24224, or the unix socket path) before piping data","Monitor fluentd and restart it before fluent-cat jobs run","Capture fluent-cat stdout: aborted records appear as '!<time>:<json>' lines you can resend","Prefer a unix socket on the same host to avoid TCP/firewall failures"],"tags":["fluentd","fluent-cat","connection","retry","tcp"],"backgroundTag":"connection-retry-exhausted","analyzedSha":"dd45c6e18dc7be33b5e5a0f0767bf46307ff5626","analyzedAt":"2026-08-21T16:22:07.332Z","schemaVersion":2},"datasetVersion":"2026-08-21T18:17:14.833Z"}