{"record":{"id":"e9b08e8111decfe1","repo":"D4Vinci/Scrapling","slug":"gzip-output-exceeds-gunzip-max-size-bytes","errorCode":null,"errorMessage":"gzip output exceeds {_GUNZIP_MAX_SIZE} bytes","messagePattern":"gzip output exceeds (.+?) bytes","errorType":"exception","errorClass":"OSError","httpStatus":null,"severity":"error","filePath":"scrapling/spiders/templates/_utils.py","lineNumber":22,"sourceCode":"from io import BytesIO\n\nfrom scrapling.core._types import Optional\n\n__all__ = [\"_decompress\"]\n\n_GZIP_MAGIC = b\"\\x1f\\x8b\"\n_GUNZIP_MAX_SIZE = 64 * 1024 * 1024  # 64 MiB cap, defends against gzip bombs\n\n\ndef _decompress(body: bytes, content_type: Optional[str]) -> bytes:\n    \"\"\"Gunzip `body` when the content-type or the magic bytes say it's gzipped, capped against gzip bombs.\"\"\"\n    if (content_type and (\"gzip\" in content_type.lower())) or (body[:2] == _GZIP_MAGIC):\n        out = bytearray()\n        with GzipFile(fileobj=BytesIO(body)) as f:\n            while chunk := f.read1(8192):\n                out.extend(chunk)\n                if len(out) > _GUNZIP_MAX_SIZE:\n                    raise OSError(f\"gzip output exceeds {_GUNZIP_MAX_SIZE} bytes\")\n        return bytes(out)\n    return body\n","sourceCodeStart":4,"sourceCodeEnd":25,"githubUrl":"https://github.com/D4Vinci/Scrapling/blob/5d213a2d4764002bfc4fed33c32fe09fa8b0bf7f/scrapling/spiders/templates/_utils.py#L4-L25","documentation":"The spider templates' response body decompressor gunzips bodies with a 64 MiB output cap (_GUNZIP_MAX_SIZE) to defend against gzip bombs. If the decompressed output exceeds the cap, an OSError is raised instead of exhausting memory.","triggerScenarios":"A fetched response arrives gzip-encoded (Content-Type contains 'gzip' or the body starts with the 1f 8b magic bytes) and decompresses to more than 64 MiB; a malicious or misconfigured server serving a gzip bomb to a template spider (feed/sitemap style).","commonSituations":"Crawling hostile or buggy servers that return huge compressed payloads; legitimately enormous feeds (large product/sitemap exports) exceeding the cap; test environments with synthetic oversized fixtures.","solutions":["If the target legitimately serves >64 MiB decompressed bodies, fetch and decompress it outside the template (custom spider with your own bounded decompression)","Skip/flag the URL on OSError and continue the crawl instead of letting it propagate","Report/block the offending domain if it is an actual gzip bomb; do not raise the cap blindly"],"exampleFix":"// before\nasync def parse(self, response):\n    body = response.body  # template decompress -> OSError on 64MiB+ gzip\n\n// after\nasync def parse(self, response):\n    try:\n        body = _decompress(response.body, \"application/gzip\")\n    except OSError:\n        self.logger.warning(f\"gzip bomb from {response.url}, skipping\")\n        return","handlingStrategy":"try-catch","validationCode":"if len(response.body) < 10 * 1024 * 1024:  # cheap pre-check on compressed size\n    body = _decompress(response.body, \"application/gzip\")","typeGuard":null,"tryCatchPattern":"except OSError as e:\n    if \"gzip output exceeds\" in str(e):\n        logger.warning(\"skipping oversized gzip payload: %s\", response.url)\n        return  # drop this response, keep crawling","preventionTips":["Treat any server that decompresses past 64 MiB as hostile until proven otherwise","Skip-and-log on OSError instead of letting it kill the crawl","Do not raise the hard cap in shared code paths"],"tags":["security","gzip","resource-limits","response-processing"],"backgroundTag":null,"analyzedSha":"5d213a2d4764002bfc4fed33c32fe09fa8b0bf7f","analyzedAt":"2026-08-14T22:23:09.440Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}