fluent/fluentd · error

failed to flush the buffer.

Error message

failed to flush the buffer.

What it means

Emitted by Fluent::Plugin::Output#log_retry_error every time a buffered chunk fails to flush to the primary output (or to the secondary when using_secondary is true) and is queued for another retry attempt. The same log line carries the underlying exception in the `error:` field plus retry_times and next_retry_time, so the root cause is the wrapped exception, not this message. Data is not lost at this point; it is only dropped later if retries are exhausted ('Hit limit for retries. dropping all chunks in the buffer queue.').

Source

Thrown at lib/fluent/plugin/output.rb:1389

            @retry.step
          else
            @retry.recalc_next_time # to prevent all flush threads from retrying at the same time
          end

          if @retry.limit?
            handle_limit_reached(error)
          elsif error
            log_retry_error(error, chunk_id_hex, using_secondary)
          end
        end
      end

      def log_retry_error(error, chunk_id_hex, using_secondary)
        return unless error
        if using_secondary
          msg = "failed to flush the buffer with secondary output."
        else
          msg = "failed to flush the buffer."
        end
        log.warn(msg, retry_times: @retry.steps, next_retry_time: @retry.next_time.round, chunk: chunk_id_hex, error: error)
        log.warn_backtrace(error.backtrace)
      end

      def handle_limit_reached(error)
        if error
          records = @buffer.queued_records
          msg = "Hit limit for retries. dropping all chunks in the buffer queue."
          log.error msg, retry_times: @retry.steps, records: records, error: error
          log.error_backtrace error.backtrace
        end
        @buffer.clear_queue!
        log.debug "buffer queue cleared"
        @retry = nil
      end

      def retry_state(randomize)

View on GitHub (pinned to dd45c6e18d)

Solutions

1. Read the `error:` field and backtrace on the same log line and fix that root cause first (connectivity, TLS, credentials, destination capacity).
2. Verify the destination is reachable and healthy from the Fluentd host (telnet/curl to the configured host:port).
3. Configure a <secondary> output (e.g. local file or alternate forward target) so chunks survive destination outages instead of being dropped after retry limits.
4. Tune retry parameters (retry_timeout, retry_max_times, retry_secondary_threshold) and buffer limits (queued_chunks_limit_size, total_limit_size) to ride out expected outage windows.
5. Confirm the buffer path is writable and has free space if using file buffer.

Example fix

# before
<match **>
  @type forward
  <server>
    host downstream.example.com
    port 24224
  </server>
  <buffer>
    @type file
    path /var/log/fluent/buffer
  </buffer>
</match>

# after (add retry resilience + secondary so flush failures do not end in dropped chunks)
<match **>
  @type forward
  <server>
    host downstream.example.com
    port 24224
  </server>
  <buffer>
    @type file
    path /var/log/fluent/buffer
    retry_timeout 72h
    retry_max_times 30
  </buffer>
  <secondary>
    @type file
    path /var/log/fluent/fallback
  </secondary>
</match>

# Pre-flight check that the forward destination accepts connections before deploying
require 'socket'
begin
  sock = TCPSocket.new('downstream.example.com', 24224)
  sock.close
  puts 'destination reachable'
rescue Errno::ECONNREFUSED, Errno::EHOSTUNREACH, SocketError => e
  puts "destination down: #{e.class} #{e.message}"
end

When it happens

Trigger: Any output plugin's write/try_write raising while a buffered chunk flushes: out_forward to an unreachable/refusing node, out_http getting non-2xx on a retryable code, TLS handshake failure, DNS resolution failure, destination disk full, or authentication failure against the destination.

Common situations: Destination Fluentd/HTTP server temporarily down during deploys, wrong port or TLS mismatch between forward peers, network partitions, full disk on the receiver, or retry_timeout/buffer queue limits sized too small for the outage duration.

Related errors

• exceed retry limit
• <buffer> section is configured, but plugin '#{self.class}' d
• secondary plugin '#{self.class}' must support buffering, but
• Invalid path component detected in #{matched}: #{replace}
• Invalid path component detected, replaced to: #{rvalue}

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/9b6e16cd381165a3. Report an issue: GitHub.