Graphify-Labs/graphify · error · OSError

Response from {url!r} exceeds size limit ({max_bytes // 1_04

Error message

Response from {url!r} exceeds size limit ({max_bytes // 1_048_576} MB). Aborting download.

What it means

Raised inside safe_fetch (graphify/security.py) while streaming the response body: after each 64 KiB read, the running total is compared against max_bytes, and exceeding it aborts the download instead of buffering an unbounded response. This is a memory-bomb defence for fetched URLs, defaulting to 10 MB (=_MAX_TEXT_BYTES) for text fetches via safe_fetch_text.

Source

Thrown at graphify/security.py:293

    opener = _build_opener()
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0 graphify/1.0"})

    with opener.open(req, timeout=timeout) as resp:
        # urllib raises HTTPError for non-2xx when using urlopen directly;
        # with a custom opener we check manually to be safe.
        status = getattr(resp, "status", None) or getattr(resp, "code", None)
        if status is not None and not (200 <= status < 300):
            raise urllib.error.HTTPError(url, status, f"HTTP {status}", {}, None)

        chunks: list[bytes] = []
        total = 0
        while True:
            chunk = resp.read(65_536)
            if not chunk:
                break
            total += len(chunk)
            if total > max_bytes:
                raise OSError(
                    f"Response from {url!r} exceeds size limit "
                    f"({max_bytes // 1_048_576} MB). Aborting download."
                )
            chunks.append(chunk)

    return b"".join(chunks)


def safe_fetch_text(url: str, max_bytes: int = _MAX_TEXT_BYTES, timeout: int = 15) -> str:
    """Fetch *url* and return decoded text (UTF-8, replacing bad bytes).

    Wraps safe_fetch with tighter defaults for HTML / text content.
    """
    raw = safe_fetch(url, max_bytes=max_bytes, timeout=timeout)
    return raw.decode("utf-8", errors="replace")


# ---------------------------------------------------------------------------

View on GitHub (pinned to 7fe58b0b0f)

Solutions

Pass an explicit larger max_bytes if you genuinely expect a big body: safe_fetch(url, max_bytes=100*1024*1024).
Pre-check Content-Length when present and skip the fetch or stream to disk yourself for very large files.
Verify you are fetching the right URL — large responses are often binaries that should not go through the text helper.
For files you control, download once out-of-band and read locally instead of repeatedly fetching through the size-capped helper.

Example fix

# before
text = safe_fetch_text("https://example.com/big-report.html")  # body > 10 MB

# after
text = safe_fetch_text("https://example.com/big-report.html", max_bytes=50_000_000)

Defensive patterns

Strategy: validation

Validate before calling

def fetch_within(url: str, max_bytes: int, timeout: int = 15) -> bytes:
    req = urllib.request.Request(url, headers={"User-Agent": "graphify-user"})
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        cl = resp.headers.get("Content-Length")
        if cl and int(cl) > max_bytes:
            raise ValueError(f"declared size {cl} exceeds budget {max_bytes}; refusing")
    return safe_fetch(url, max_bytes=max_bytes, timeout=timeout)

Type guard

def url_within_budget(url: str, max_bytes: int) -> bool:
    with urllib.request.urlopen(urllib.request.Request(url, method="HEAD"), timeout=10) as r:
        cl = r.headers.get("Content-Length")
    return cl is None or int(cl) <= max_bytes

Try / catch

try:
    text = safe_fetch_text(url)
except OSError as e:
    if "exceeds size limit" in str(e):
        text = safe_fetch_text(url, max_bytes=100_000_000)  # explicit, deliberate raise
    else:
        raise

Prevention

Decide the expected body size before fetching and pass max_bytes explicitly.
Use HEAD/Content-Length pre-checks for URLs of unknown size.
Route large downloads to disk streaming, not through the size-capped text helper.

When it happens

Trigger: Calling safe_fetch(url, max_bytes=N) or safe_fetch_text(url) where the server returns a body larger than the limit (default 10 MB for text). The error fires mid-stream after roughly max_bytes + 64KiB have been read.

Common situations: Fetching a large artifact/CDN page that is actually a binary or a huge HTML bundle; a redirect landing on a big file; a documentation URL that returns an uncompressed multi-megabyte page; using safe_fetch_text on a dataset URL by mistake.

Related errors

AI-assisted analysis of Graphify-Labs/graphify@7fe58b0b0f (2026-08-14). Data as JSON: /api/errors/b37666ea52fd019e. Report an issue: GitHub.