fluent/fluentd · error · Fluent::Plugin::Buffer::BufferOverflowError

buffer space has too many data

Error message

buffer space has too many data

What it means

Buffer#write raises BufferOverflowError when storable? is false, i.e. total_limit_size is no longer greater than the sum of staged (@stage_size_metrics) and queued (@queue_size_metrics) bytes. This is the file/memory buffer backpressure signal: the output cannot flush chunks as fast as events arrive, so the buffer is full and new writes are rejected. The owning output plugin catches this and applies its overflow_action (throw_exception, block, or drop_oldest_chunk).

Source

Thrown at lib/fluent/plugin/buffer.rb:335

      def new_metadata(timekey: nil, tag: nil, variables: nil)
        Metadata.new(timekey, tag, variables)
      end

      # Keep this method for existing code
      def metadata(timekey: nil, tag: nil, variables: nil)
        Metadata.new(timekey, tag, variables)
      end

      def timekeys
        @timekeys.keys
      end

      # metadata MUST have consistent object_id for each variation
      # data MUST be Array of serialized events, or EventStream
      # metadata_and_data MUST be a hash of { metadata => data }
      def write(metadata_and_data, format: nil, size: nil, enqueue: false)
        return if metadata_and_data.size < 1
        raise BufferOverflowError, "buffer space has too many data" unless storable?

        log.on_trace { log.trace "writing events into buffer", instance: self.object_id, metadata_size: metadata_and_data.size }

        operated_chunks = []
        unstaged_chunks = {} # metadata => [chunk, chunk, ...]
        chunks_to_enqueue = []
        staged_bytesizes_by_chunk = {}
        # track internal BufferChunkOverflowError in write_step_by_step
        buffer_chunk_overflow_errors = []

        begin
          # sort metadata to get lock of chunks in same order with other threads
          metadata_and_data.keys.sort.each do |metadata|
            data = metadata_and_data[metadata]
            write_once(metadata, data, format: format, size: size) do |chunk, adding_bytesize, error|
              chunk.mon_enter # add lock to prevent to be committed/rollbacked from other threads
              operated_chunks << chunk
              if chunk.staged?

View on GitHub (pinned to dd45c6e18d)

Solutions

1. Fix or speed up the downstream destination so the queue drains (this is the root cause — the buffer is only the symptom)
2. Increase total_limit_size in the <buffer> section to ride out longer outages (ensure the disk actually has that much free space)
3. Tune flush parameters: lower flush_interval, lower chunk_limit_size so smaller chunks flush sooner, enable flush_thread_count > 1
4. Set overflow_action explicitly: block to apply backpressure to inputs, or drop_oldest_chunk to trade data for uptime (default throw_exception will surface the error to the input, e.g. in_tail pauses)
5. Add a <secondary> output so failed flushes are diverted instead of retrying forever

Example fix

# before
<buffer>
  @type file
  total_limit_size 512MB
</buffer>

# after
<buffer>
  @type file
  total_limit_size 8GB
  chunk_limit_size 8MB
  flush_thread_count 4
  overflow_action block
</buffer>

# Expose the same predicate fluentd uses before writes
def buffer_storable?(output)
  buf = output.instance_variable_get(:@buffer) or return true
  total = buf.buffer_config.total_limit_size
  used = Fluent::Plugin::Buffer::StageSizeMetrics # monitor via metrics plugin instead:
  # attach <metrics> metric.rb or prometheus and alert on fluentd_buffer_total_bytes/total_limit_size > 0.8
  true
end

# Ops-level: alert when buffer byte utilization exceeds 80%
# (prometheus input exposes stage/queue length and bytesize)

begin
  output.emit_events(tag, es)
rescue Fluent::Plugin::Buffer::BufferOverflowError
  case overflow_action
  when :block then retry_after_backoff      # let input backpressure handle it
  when :drop_oldest_chunk then retry_once   # space was freed
  else raise                                 # throw_exception: surface to input
  end
end

When it happens

Trigger: An output plugin with a file/file_single/memory buffer whose downstream destination is slow or down (network outage, forward peer unreachable, HTTP endpoint timing out) so queued chunks accumulate past total_limit_size (default 64GB for file_single, 512MB for memory); also a too-small total_limit_size combined with a burst of events, or retry_timeout/retry_max_times keeping chunks in queue while new data streams in.

Common situations: Forward output to a distant aggregator that goes down over the weekend; disk-backed buffer on a slow disk with large chunk_limit_size so chunks enqueue slowly; undersized total_limit_size after switching buffer type (memory 512MB default vs file 64GB); in_tail input reading a huge backlog into a stalled output.

Related errors

• failed to flush the buffer with secondary output.
• this plugin '#{self.class}' cannot handle arguments for <buf
• this plugin '#{self.class}' allows <buffer tag> only
• time_slice_format only with %Y or %m is too long
• buffer path is not configured. specify 'path' in <buffer>

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/1855d517a1ad5eaf. Report an issue: GitHub.