{"record":{"id":"1855d517a1ad5eaf","repo":"fluent/fluentd","slug":"buffer-space-has-too-many-data","errorCode":null,"errorMessage":"buffer space has too many data","messagePattern":"buffer space has too many data","errorType":"exception","errorClass":"Fluent::Plugin::Buffer::BufferOverflowError","httpStatus":null,"severity":"error","filePath":"lib/fluent/plugin/buffer.rb","lineNumber":335,"sourceCode":"      def new_metadata(timekey: nil, tag: nil, variables: nil)\n        Metadata.new(timekey, tag, variables)\n      end\n\n      # Keep this method for existing code\n      def metadata(timekey: nil, tag: nil, variables: nil)\n        Metadata.new(timekey, tag, variables)\n      end\n\n      def timekeys\n        @timekeys.keys\n      end\n\n      # metadata MUST have consistent object_id for each variation\n      # data MUST be Array of serialized events, or EventStream\n      # metadata_and_data MUST be a hash of { metadata => data }\n      def write(metadata_and_data, format: nil, size: nil, enqueue: false)\n        return if metadata_and_data.size < 1\n        raise BufferOverflowError, \"buffer space has too many data\" unless storable?\n\n        log.on_trace { log.trace \"writing events into buffer\", instance: self.object_id, metadata_size: metadata_and_data.size }\n\n        operated_chunks = []\n        unstaged_chunks = {} # metadata => [chunk, chunk, ...]\n        chunks_to_enqueue = []\n        staged_bytesizes_by_chunk = {}\n        # track internal BufferChunkOverflowError in write_step_by_step\n        buffer_chunk_overflow_errors = []\n\n        begin\n          # sort metadata to get lock of chunks in same order with other threads\n          metadata_and_data.keys.sort.each do |metadata|\n            data = metadata_and_data[metadata]\n            write_once(metadata, data, format: format, size: size) do |chunk, adding_bytesize, error|\n              chunk.mon_enter # add lock to prevent to be committed/rollbacked from other threads\n              operated_chunks << chunk\n              if chunk.staged?","sourceCodeStart":317,"sourceCodeEnd":353,"githubUrl":"https://github.com/fluent/fluentd/blob/dd45c6e18dc7be33b5e5a0f0767bf46307ff5626/lib/fluent/plugin/buffer.rb#L317-L353","documentation":"Buffer#write raises BufferOverflowError when storable? is false, i.e. total_limit_size is no longer greater than the sum of staged (@stage_size_metrics) and queued (@queue_size_metrics) bytes. This is the file/memory buffer backpressure signal: the output cannot flush chunks as fast as events arrive, so the buffer is full and new writes are rejected. The owning output plugin catches this and applies its overflow_action (throw_exception, block, or drop_oldest_chunk).","triggerScenarios":"An output plugin with a file/file_single/memory buffer whose downstream destination is slow or down (network outage, forward peer unreachable, HTTP endpoint timing out) so queued chunks accumulate past total_limit_size (default 64GB for file_single, 512MB for memory); also a too-small total_limit_size combined with a burst of events, or retry_timeout/retry_max_times keeping chunks in queue while new data streams in.","commonSituations":"Forward output to a distant aggregator that goes down over the weekend; disk-backed buffer on a slow disk with large chunk_limit_size so chunks enqueue slowly; undersized total_limit_size after switching buffer type (memory 512MB default vs file 64GB); in_tail input reading a huge backlog into a stalled output.","solutions":["Fix or speed up the downstream destination so the queue drains (this is the root cause — the buffer is only the symptom)","Increase total_limit_size in the <buffer> section to ride out longer outages (ensure the disk actually has that much free space)","Tune flush parameters: lower flush_interval, lower chunk_limit_size so smaller chunks flush sooner, enable flush_thread_count > 1","Set overflow_action explicitly: block to apply backpressure to inputs, or drop_oldest_chunk to trade data for uptime (default throw_exception will surface the error to the input, e.g. in_tail pauses)","Add a <secondary> output so failed flushes are diverted instead of retrying forever"],"exampleFix":"# before\n<buffer>\n  @type file\n  total_limit_size 512MB\n</buffer>\n\n# after\n<buffer>\n  @type file\n  total_limit_size 8GB\n  chunk_limit_size 8MB\n  flush_thread_count 4\n  overflow_action block\n</buffer>","handlingStrategy":"retry","validationCode":"# Expose the same predicate fluentd uses before writes\ndef buffer_storable?(output)\n  buf = output.instance_variable_get(:@buffer) or return true\n  total = buf.buffer_config.total_limit_size\n  used = Fluent::Plugin::Buffer::StageSizeMetrics # monitor via metrics plugin instead:\n  # attach <metrics> metric.rb or prometheus and alert on fluentd_buffer_total_bytes/total_limit_size > 0.8\n  true\nend\n\n# Ops-level: alert when buffer byte utilization exceeds 80%\n# (prometheus input exposes stage/queue length and bytesize)","typeGuard":null,"tryCatchPattern":"begin\n  output.emit_events(tag, es)\nrescue Fluent::Plugin::Buffer::BufferOverflowError\n  case overflow_action\n  when :block then retry_after_backoff      # let input backpressure handle it\n  when :drop_oldest_chunk then retry_once   # space was freed\n  else raise                                 # throw_exception: surface to input\n  end\nend","preventionTips":["Size total_limit_size below real disk capacity and expected outage duration","Monitor stage/queue bytesize with the prometheus/metrics plugins and alert before 100%","Tune flush_thread_count, flush_interval, chunk_limit_size to keep the queue draining","Configure overflow_action deliberately and add a <secondary> output"],"tags":["fluentd","buffer-overflow","backpressure","buffer","runtime"],"backgroundTag":"buffer-backpressure-full","analyzedSha":"dd45c6e18dc7be33b5e5a0f0767bf46307ff5626","analyzedAt":"2026-08-21T16:22:07.332Z","schemaVersion":2},"datasetVersion":"2026-08-21T18:17:14.833Z"}