fluent/fluentd · error · SizeLimitError

Decompressed data exceeds limit of #{limit} bytes

Error message

Decompressed data exceeds limit of #{limit} bytes

What it means

Fluent::Plugin::Extractor.decompress_gzip(compressed_data, limit:) inflates gzip data in 64KB blocks (handling multi-member streams) and raises SizeLimitError once the output exceeds limit bytes. SizeLimitError derives from Fluent::UnrecoverableError, so retrying will not help by design. Callers pass their decompression_size_limit: in_http for gzip Content-Encoding request bodies and in_forward/CompressedMessagePackEventStream for compressed forward traffic (both default 256MB), plus chunk reads from buffers configured with compress gzip.

Source

Thrown at lib/fluent/plugin/extractor.rb:38

require 'fluent/error'

module Fluent
  module Plugin
    module Extractor
      class SizeLimitError < UnrecoverableError; end

      BYTES_TO_READ = 64 * 1024
      INFLATE_BYTES_TO_READ = 1024

      def self.decompress_gzip(compressed_data, limit:)
        io = StringIO.new(compressed_data)
        out = ''
        loop do
          reader = Zlib::GzipReader.new(io)
          while (chunk = reader.read(BYTES_TO_READ))
            out << chunk
            if out.bytesize > limit
              raise SizeLimitError, "Decompressed data exceeds limit of #{limit} bytes"
            end
          end

          unused = reader.unused
          reader.finish
          unless unused.nil?
            adjust = unused.length
            io.pos -= adjust
          end
          break if io.eof?
        end
        out
      end

      def self.decompress_zstd(compressed_data, limit:)
        io = StringIO.new(compressed_data)
        reader = Zstd::StreamReader.new(io)
        out = ''

View on GitHub (pinned to dd45c6e18d)

Solutions

1. If the payloads are legitimate, raise the receiving limit: decompression_size_limit 512MB on in_http/in_forward or inside <buffer>.
2. Fix the sender: compress smaller batches, or split the stream so single payloads stay under the limit.
3. If the limit is intentionally a protection against decompression bombs, keep it and drop/alert on SizeLimitError instead of raising it (it is UnrecoverableError; never retry).
4. For huge legitimate volumes, prefer the forward protocol with streaming or multiple smaller messages over one giant gzip body.

Example fix

# before
<source>
  @type http
  port 9880
  # gzip bodies >256MB inflated raise SizeLimitError
</source>
# after
<source>
  @type http
  port 9880
  decompression_size_limit 512MB
</source>

# before decompressing untrusted data yourself, sanity-check the ratio
raise ArgumentError, 'implausible compression ratio' if compressed.bytesize > 0 && expected_max && compressed.bytesize * 64 > expected_max * 1024
# for in_http/in_forward/buffer: set decompression_size_limit explicitly to the largest legitimate inflated size

->(data) { data.is_a?(String) && data.encoding == Encoding::ASCII_8BIT }

begin
  data = Fluent::Plugin::Extractor.decompress_gzip(body, limit: 256 * 1024 * 1024)
rescue Fluent::Plugin::Extractor::SizeLimitError
  # UnrecoverableError subclass: never retry; drop, respond 4xx, and alert
  log.warn 'gzip payload exceeded decompression_size_limit', bytes: body.bytesize
end

When it happens

Trigger: An HTTP POST with Content-Encoding: gzip to in_http whose inflated body exceeds the plugin's decompression_size_limit; a forward sender compressing entries beyond in_forward's limit; or reading a compressed buffer chunk whose content exceeds the buffer's decompression_size_limit.

Common situations: Clients switching to gzip-compressed batches that legitimately exceed 256MB inflated; batch-size growth after adding fields/hosts; malicious or misconfigured senders (zip-bomb style payloads); explicitly lowered limits after a security review.

Related errors

• staged meta file is broken. #{e.message}
• enqueued file chunk is empty
• enqueued meta file is broken. #{e.message}
• Invalid chunk found. unique_id and key not exist: #{@path}
• can't create buffer file for #{path}. Stop creating buffer f

AI-assisted analysis of fluent/fluentd@dd45c6e18d (2026-08-21). Data as JSON: /api/errors/faceba7aff584de7. Report an issue: GitHub.