fluent/fluentd · critical · BufferOverflowError

can't create buffer file for #{path}. Stop creating buffer f

Error message

can't create buffer file for #{path}. Stop creating buffer files: error = #{e}

What it means

FileChunk#create_new_chunk opens a new staged chunk data file with File.open(path,'wb+',perm); any raised StandardError is converted to BufferOverflowError with this message. The code comment states the assumption is a recoverable 'Too many open files' (EMFILE) condition — raising BufferOverflowError lets the output's overflow_action machinery (retry/block/drop) deal with it rather than crashing the agent.

Source

Thrown at lib/fluent/plugin/buffer/file_chunk.rb:293

          file.set_encoding(Encoding::ASCII_8BIT)
          file.sync = true
          file.binmode
          file.pos = pos
          callback.call(file) if callback
        end

        def create_new_chunk(path, perm)
          @path = self.class.generate_stage_chunk_path(path, @unique_id)
          @meta_path = @path + '.meta'
          begin
            @chunk = File.open(@path, 'wb+', perm)
            @chunk.set_encoding(Encoding::ASCII_8BIT)
            @chunk.sync = true
            @chunk.binmode
          rescue => e
            # Here assumes "Too many open files" like recoverable error so raising BufferOverflowError.
            # If other cases are possible, we will change error handling with proper classes.
            raise BufferOverflowError, "can't create buffer file for #{path}. Stop creating buffer files: error = #{e}"
          end
          begin
            @meta = File.open(@meta_path, 'wb', perm)
            @meta.set_encoding(Encoding::ASCII_8BIT)
            @meta.sync = true
            @meta.binmode
            write_metadata(update: false)
          rescue => e
            # This case is easier than enqueued!. Just removing pre-create buffer file
            @chunk.close rescue nil
            File.unlink(@path) rescue nil

            if @meta
              # ensure to unlink when #write_metadata fails
              @meta.close rescue nil
              File.unlink(@meta_path) rescue nil
            end

View on GitHub (pinned to dd45c6e18d)

Solutions

1. Check the trailing 'error = ...' in the message: Errno::EMFILE → raise the fd limit (ulimit -n / LimitNOFILE) ; Errno::ENOSPC → free disk or lower total_limit_size; Errno::EACCES → fix dir ownership/permissions
2. Reduce concurrent open files: fewer flush threads, fewer separate buffer paths, larger chunk_limit_size so fewer chunks exist
3. Verify with ls /proc/$(pidof fluentd)/fd | wc -l against ulimit -n
4. If it is a genuine FD leak in a plugin, capture lsof output over time and fix the leak

Example fix

# systemd: raise fd limit for fluentd
# /etc/systemd/system/fluentd.service
[Service]
LimitNOFILE=65536

# shell check before/while running
ulimit -n
ls /proc/$(pgrep -f 'fluentd.*ruby')/fd | wc -l

# Pre-start: verify fd headroom and writable buffer dir
limits = Process.getrlimit(:NOFILE)
abort "NOFILE too low: #{limits[0]}" if limits[0] < 65_536
buf_dir = File.dirname(buffer_path)
test = File.join(buf_dir, ".fd_probe_#{Process.pid}")
File.open(test,'wb'){|f| f.write('x')} ; File.unlink(test)
# runtime: watch fd usage
fds = Dir.glob("/proc/#{Process.pid}/fd").size
abort "fd usage #{fds} near limit #{limits[0]}" if fds > limits[0] * 0.8

begin
  buffer.write(metadata => data)
rescue Fluent::Plugin::Buffer::BufferOverflowError => e
  if e.message.include?("can't create buffer file")
    sleep 1 and retry   # EMFILE/ENOSPC class: transient; fix ulimit/disk while retrying
  else
    raise               # genuine buffer-full: apply overflow_action
  end
end

When it happens

Trigger: Process file-descriptor exhaustion (ulimit -n reached) when creating buffer chunk files — many buffers, many flush threads, or an FD leak; also any IO error opening the file such as ENOSPC (disk full), EACCES (permissions on the buffer dir), or ENAMETOOLONG.

Common situations: High-throughput fluentd with several file/file_single buffers hitting the default 1024 FD limit; buffer volume filled (ENOSPC); dir_permission/file_permission mismatch after a chmod; long-running FD leak in a custom plugin.

Related errors

• can't create buffer metadata for #{path}. Stop creating buff
• invalid meta data
• invalid unique_id
• invalid created_at
• invalid modified_at

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/9f847b7098bd43a0. Report an issue: GitHub.