{"record":{"id":"b37666ea52fd019e","repo":"Graphify-Labs/graphify","slug":"response-from-url-r-exceeds-size-limit-max-byt","errorCode":null,"errorMessage":"Response from {url!r} exceeds size limit ({max_bytes // 1_048_576} MB). Aborting download.","messagePattern":"Response from (.+?) exceeds size limit \\((.+?) MB\\)\\. Aborting download\\.","errorType":"exception","errorClass":"OSError","httpStatus":null,"severity":"error","filePath":"graphify/security.py","lineNumber":293,"sourceCode":"    opener = _build_opener()\n    req = urllib.request.Request(url, headers={\"User-Agent\": \"Mozilla/5.0 graphify/1.0\"})\n\n    with opener.open(req, timeout=timeout) as resp:\n        # urllib raises HTTPError for non-2xx when using urlopen directly;\n        # with a custom opener we check manually to be safe.\n        status = getattr(resp, \"status\", None) or getattr(resp, \"code\", None)\n        if status is not None and not (200 <= status < 300):\n            raise urllib.error.HTTPError(url, status, f\"HTTP {status}\", {}, None)\n\n        chunks: list[bytes] = []\n        total = 0\n        while True:\n            chunk = resp.read(65_536)\n            if not chunk:\n                break\n            total += len(chunk)\n            if total > max_bytes:\n                raise OSError(\n                    f\"Response from {url!r} exceeds size limit \"\n                    f\"({max_bytes // 1_048_576} MB). Aborting download.\"\n                )\n            chunks.append(chunk)\n\n    return b\"\".join(chunks)\n\n\ndef safe_fetch_text(url: str, max_bytes: int = _MAX_TEXT_BYTES, timeout: int = 15) -> str:\n    \"\"\"Fetch *url* and return decoded text (UTF-8, replacing bad bytes).\n\n    Wraps safe_fetch with tighter defaults for HTML / text content.\n    \"\"\"\n    raw = safe_fetch(url, max_bytes=max_bytes, timeout=timeout)\n    return raw.decode(\"utf-8\", errors=\"replace\")\n\n\n# ---------------------------------------------------------------------------","sourceCodeStart":275,"sourceCodeEnd":311,"githubUrl":"https://github.com/Graphify-Labs/graphify/blob/7fe58b0b0f3873be9a21c30106b8b8527c353aa6/graphify/security.py#L275-L311","documentation":"Raised inside safe_fetch (graphify/security.py) while streaming the response body: after each 64 KiB read, the running total is compared against max_bytes, and exceeding it aborts the download instead of buffering an unbounded response. This is a memory-bomb defence for fetched URLs, defaulting to 10 MB (=_MAX_TEXT_BYTES) for text fetches via safe_fetch_text.","triggerScenarios":"Calling safe_fetch(url, max_bytes=N) or safe_fetch_text(url) where the server returns a body larger than the limit (default 10 MB for text). The error fires mid-stream after roughly max_bytes + 64KiB have been read.","commonSituations":"Fetching a large artifact/CDN page that is actually a binary or a huge HTML bundle; a redirect landing on a big file; a documentation URL that returns an uncompressed multi-megabyte page; using safe_fetch_text on a dataset URL by mistake.","solutions":["Pass an explicit larger max_bytes if you genuinely expect a big body: safe_fetch(url, max_bytes=100*1024*1024).","Pre-check Content-Length when present and skip the fetch or stream to disk yourself for very large files.","Verify you are fetching the right URL — large responses are often binaries that should not go through the text helper.","For files you control, download once out-of-band and read locally instead of repeatedly fetching through the size-capped helper."],"exampleFix":"# before\ntext = safe_fetch_text(\"https://example.com/big-report.html\")  # body > 10 MB\n\n# after\ntext = safe_fetch_text(\"https://example.com/big-report.html\", max_bytes=50_000_000)","handlingStrategy":"validation","validationCode":"def fetch_within(url: str, max_bytes: int, timeout: int = 15) -> bytes:\n    req = urllib.request.Request(url, headers={\"User-Agent\": \"graphify-user\"})\n    with urllib.request.urlopen(req, timeout=timeout) as resp:\n        cl = resp.headers.get(\"Content-Length\")\n        if cl and int(cl) > max_bytes:\n            raise ValueError(f\"declared size {cl} exceeds budget {max_bytes}; refusing\")\n    return safe_fetch(url, max_bytes=max_bytes, timeout=timeout)","typeGuard":"def url_within_budget(url: str, max_bytes: int) -> bool:\n    with urllib.request.urlopen(urllib.request.Request(url, method=\"HEAD\"), timeout=10) as r:\n        cl = r.headers.get(\"Content-Length\")\n    return cl is None or int(cl) <= max_bytes","tryCatchPattern":"try:\n    text = safe_fetch_text(url)\nexcept OSError as e:\n    if \"exceeds size limit\" in str(e):\n        text = safe_fetch_text(url, max_bytes=100_000_000)  # explicit, deliberate raise\n    else:\n        raise","preventionTips":["Decide the expected body size before fetching and pass max_bytes explicitly.","Use HEAD/Content-Length pre-checks for URLs of unknown size.","Route large downloads to disk streaming, not through the size-capped text helper."],"tags":["network","limits","memory","fetch"],"backgroundTag":null,"analyzedSha":"7fe58b0b0f3873be9a21c30106b8b8527c353aa6","analyzedAt":"2026-08-14T19:23:21.323Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}