fluent/fluentd · warning

failed to flush the buffer with secondary output.

Error message

failed to flush the buffer with secondary output.

What it means

This is not a raised exception but a log.warn emitted by BufferedOutput#log_retry_error (lib/fluent/plugin/output.rb:1387) via update_retry_state: a buffered chunk failed to flush, retries remain, AND the attempt already fell through to the <secondary> output. Seeing it means the primary destination has been failing long enough to exhaust the primary retry budget and secondary writes are now also erroring — data is still queued and will keep retrying, but the delivery path is degraded on both tiers.

Source

Thrown at lib/fluent/plugin/output.rb:1387

          # @retry.step is called almost as many times as the number of flush threads in a short time.
          if Time.now >= @retry.next_time
            @retry.step
          else
            @retry.recalc_next_time # to prevent all flush threads from retrying at the same time
          end

          if @retry.limit?
            handle_limit_reached(error)
          elsif error
            log_retry_error(error, chunk_id_hex, using_secondary)
          end
        end
      end

      def log_retry_error(error, chunk_id_hex, using_secondary)
        return unless error
        if using_secondary
          msg = "failed to flush the buffer with secondary output."
        else
          msg = "failed to flush the buffer."
        end
        log.warn(msg, retry_times: @retry.steps, next_retry_time: @retry.next_time.round, chunk: chunk_id_hex, error: error)
        log.warn_backtrace(error.backtrace)
      end

      def handle_limit_reached(error)
        if error
          records = @buffer.queued_records
          msg = "Hit limit for retries. dropping all chunks in the buffer queue."
          log.error msg, retry_times: @retry.steps, records: records, error: error
          log.error_backtrace error.backtrace
        end
        @buffer.clear_queue!
        log.debug "buffer queue cleared"
        @retry = nil
      end

View on GitHub (pinned to dd45c6e18d)

Solutions

1. Read the attached 'error' field in the same log line — it identifies the secondary's actual failure (disk full, auth, network)
2. Restore the secondary first (it is the safety net): free disk space, fix credentials/permissions on the secondary target
3. Then fix the primary destination and let retries drain the queue naturally; verify chunks decrease with fluentd's buffer metrics / log
4. Tune <buffer> retry_timeout and secondary_threshold if secondary activation is premature, and check retry_limit so handle_limit_reached (drop-all) never fires
5. Monitor buffer queued records/total_size and alert before the queue approaches total_limit_size to avoid eventual chunk loss

Example fix

# before
<match app.**>
  @type forward
  <buffer>
    retry_timeout 30s
  </buffer>
  <secondary>
    @type file
    path /var/log/fluent/backup   # disk full -> secondary also fails
  </secondary>
</match>

# after
<match app.**>
  @type forward
  <buffer>
    retry_timeout 72h
    retry_limit 30
  </buffer>
  <secondary>
    @type file
    path /mnt/bigvolume/fluent/backup   # writable volume with capacity alerts
  </secondary>
</match>

# monitoring pre-check (not error prevention, but early warning):
# fluentd-prometheus metrics: fluentd_output_status_buffer_total_bytes,
# fluentd_output_status_retry_count, fluentd_output_status_buffer_queue_length
# alert when queued bytes > 60% of total_limit_size or retry_count > 0 for 10m

When it happens

Trigger: An output with a <secondary> block (e.g. primary @type forward to a collector, secondary @type file to local spool) where the primary exceeded its retry_timeout and switched to secondary, then the secondary write raised too; update_retry_state is called with using_secondary=true and a non-nil error, logging retry_times, next_retry_time, chunk id and the underlying error.

Common situations: Remote collector down AND local secondary disk is full/read-only; S3 primary failing on credentials while the secondary archive target also rejects writes; network partition plus permission drift on the fallback path; retry_timeout/secondary_threshold tuned so low that every hiccup skips to secondary.

Related errors

• buffer space has too many data
• exceed retry limit
• this plugin '#{self.class}' cannot handle arguments for <buf
• this plugin '#{self.class}' allows <buffer tag> only
• time_slice_format only with %Y or %m is too long

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/576ff7a3c84353cd. Report an issue: GitHub.