D4Vinci/Scrapling · error · OSError

gzip output exceeds {_GUNZIP_MAX_SIZE} bytes

Error message

gzip output exceeds {_GUNZIP_MAX_SIZE} bytes

What it means

The spider templates' response body decompressor gunzips bodies with a 64 MiB output cap (_GUNZIP_MAX_SIZE) to defend against gzip bombs. If the decompressed output exceeds the cap, an OSError is raised instead of exhausting memory.

Source

Thrown at scrapling/spiders/templates/_utils.py:22

from io import BytesIO

from scrapling.core._types import Optional

__all__ = ["_decompress"]

_GZIP_MAGIC = b"\x1f\x8b"
_GUNZIP_MAX_SIZE = 64 * 1024 * 1024  # 64 MiB cap, defends against gzip bombs


def _decompress(body: bytes, content_type: Optional[str]) -> bytes:
    """Gunzip `body` when the content-type or the magic bytes say it's gzipped, capped against gzip bombs."""
    if (content_type and ("gzip" in content_type.lower())) or (body[:2] == _GZIP_MAGIC):
        out = bytearray()
        with GzipFile(fileobj=BytesIO(body)) as f:
            while chunk := f.read1(8192):
                out.extend(chunk)
                if len(out) > _GUNZIP_MAX_SIZE:
                    raise OSError(f"gzip output exceeds {_GUNZIP_MAX_SIZE} bytes")
        return bytes(out)
    return body

View on GitHub (pinned to 5d213a2d47)

Solutions

  1. If the target legitimately serves >64 MiB decompressed bodies, fetch and decompress it outside the template (custom spider with your own bounded decompression)
  2. Skip/flag the URL on OSError and continue the crawl instead of letting it propagate
  3. Report/block the offending domain if it is an actual gzip bomb; do not raise the cap blindly

Example fix

// before
async def parse(self, response):
    body = response.body  # template decompress -> OSError on 64MiB+ gzip

// after
async def parse(self, response):
    try:
        body = _decompress(response.body, "application/gzip")
    except OSError:
        self.logger.warning(f"gzip bomb from {response.url}, skipping")
        return
Defensive patterns

Strategy: try-catch

Validate before calling

if len(response.body) < 10 * 1024 * 1024:  # cheap pre-check on compressed size
    body = _decompress(response.body, "application/gzip")

Try / catch

except OSError as e:
    if "gzip output exceeds" in str(e):
        logger.warning("skipping oversized gzip payload: %s", response.url)
        return  # drop this response, keep crawling

Prevention

When it happens

Trigger: A fetched response arrives gzip-encoded (Content-Type contains 'gzip' or the body starts with the 1f 8b magic bytes) and decompresses to more than 64 MiB; a malicious or misconfigured server serving a gzip bomb to a template spider (feed/sitemap style).

Common situations: Crawling hostile or buggy servers that return huge compressed payloads; legitimately enormous feeds (large product/sitemap exports) exceeding the cap; test environments with synthetic oversized fixtures.

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/e9b08e8111decfe1. Report an issue: GitHub.