{"record":{"id":"faceba7aff584de7","repo":"fluent/fluentd","slug":"decompressed-data-exceeds-limit-of-limit-bytes","errorCode":null,"errorMessage":"Decompressed data exceeds limit of #{limit} bytes","messagePattern":"Decompressed data exceeds limit of #(.+?) bytes","errorType":"exception","errorClass":"SizeLimitError","httpStatus":null,"severity":"error","filePath":"lib/fluent/plugin/extractor.rb","lineNumber":38,"sourceCode":"require 'fluent/error'\n\nmodule Fluent\n  module Plugin\n    module Extractor\n      class SizeLimitError < UnrecoverableError; end\n\n      BYTES_TO_READ = 64 * 1024\n      INFLATE_BYTES_TO_READ = 1024\n\n      def self.decompress_gzip(compressed_data, limit:)\n        io = StringIO.new(compressed_data)\n        out = ''\n        loop do\n          reader = Zlib::GzipReader.new(io)\n          while (chunk = reader.read(BYTES_TO_READ))\n            out << chunk\n            if out.bytesize > limit\n              raise SizeLimitError, \"Decompressed data exceeds limit of #{limit} bytes\"\n            end\n          end\n\n          unused = reader.unused\n          reader.finish\n          unless unused.nil?\n            adjust = unused.length\n            io.pos -= adjust\n          end\n          break if io.eof?\n        end\n        out\n      end\n\n      def self.decompress_zstd(compressed_data, limit:)\n        io = StringIO.new(compressed_data)\n        reader = Zstd::StreamReader.new(io)\n        out = ''","sourceCodeStart":20,"sourceCodeEnd":56,"githubUrl":"https://github.com/fluent/fluentd/blob/dd45c6e18dc7be33b5e5a0f0767bf46307ff5626/lib/fluent/plugin/extractor.rb#L20-L56","documentation":"Fluent::Plugin::Extractor.decompress_gzip(compressed_data, limit:) inflates gzip data in 64KB blocks (handling multi-member streams) and raises SizeLimitError once the output exceeds limit bytes. SizeLimitError derives from Fluent::UnrecoverableError, so retrying will not help by design. Callers pass their decompression_size_limit: in_http for gzip Content-Encoding request bodies and in_forward/CompressedMessagePackEventStream for compressed forward traffic (both default 256MB), plus chunk reads from buffers configured with compress gzip.","triggerScenarios":"An HTTP POST with Content-Encoding: gzip to in_http whose inflated body exceeds the plugin's decompression_size_limit; a forward sender compressing entries beyond in_forward's limit; or reading a compressed buffer chunk whose content exceeds the buffer's decompression_size_limit.","commonSituations":"Clients switching to gzip-compressed batches that legitimately exceed 256MB inflated; batch-size growth after adding fields/hosts; malicious or misconfigured senders (zip-bomb style payloads); explicitly lowered limits after a security review.","solutions":["If the payloads are legitimate, raise the receiving limit: decompression_size_limit 512MB on in_http/in_forward or inside <buffer>.","Fix the sender: compress smaller batches, or split the stream so single payloads stay under the limit.","If the limit is intentionally a protection against decompression bombs, keep it and drop/alert on SizeLimitError instead of raising it (it is UnrecoverableError; never retry).","For huge legitimate volumes, prefer the forward protocol with streaming or multiple smaller messages over one giant gzip body."],"exampleFix":"# before\n<source>\n  @type http\n  port 9880\n  # gzip bodies >256MB inflated raise SizeLimitError\n</source>\n# after\n<source>\n  @type http\n  port 9880\n  decompression_size_limit 512MB\n</source>","handlingStrategy":"try-catch","validationCode":"# before decompressing untrusted data yourself, sanity-check the ratio\nraise ArgumentError, 'implausible compression ratio' if compressed.bytesize > 0 && expected_max && compressed.bytesize * 64 > expected_max * 1024\n# for in_http/in_forward/buffer: set decompression_size_limit explicitly to the largest legitimate inflated size","typeGuard":"->(data) { data.is_a?(String) && data.encoding == Encoding::ASCII_8BIT }","tryCatchPattern":"begin\n  data = Fluent::Plugin::Extractor.decompress_gzip(body, limit: 256 * 1024 * 1024)\nrescue Fluent::Plugin::Extractor::SizeLimitError\n  # UnrecoverableError subclass: never retry; drop, respond 4xx, and alert\n  log.warn 'gzip payload exceeded decompression_size_limit', bytes: body.bytesize\nend","preventionTips":["Set decompression_size_limit explicitly on in_http/in_forward instead of relying on the 256MB default.","Size sender batches so inflated output stays comfortably under the limit.","Treat SizeLimitError as a security signal on untrusted endpoints.","Prefer streaming/forward protocols over single giant compressed bodies."],"tags":["fluentd","ruby","gzip","decompression","size-limit","zip-bomb"],"backgroundTag":"decompression-size-limit-exceeded","analyzedSha":"dd45c6e18dc7be33b5e5a0f0767bf46307ff5626","analyzedAt":"2026-08-21T16:22:07.332Z","schemaVersion":2},"datasetVersion":"2026-08-21T18:17:14.833Z"}