{"record":{"id":"7949a5270fd37d80","repo":"crewAIInc/crewAI","slug":"response-body-from-response-url-exceeds-the-m","errorCode":null,"errorMessage":"Response body from '{response.url}' exceeds the {max_bytes} byte limit.","messagePattern":"Response body from '(.+?)' exceeds the (.+?) byte limit\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/security/safe_requests.py","lineNumber":157,"sourceCode":"        url,\n        max_redirects=max_redirects,\n        headers=headers,\n        timeout=timeout,\n        stream=True,\n    )\n    try:\n        response.raise_for_status()\n\n        chunks: list[bytes] = []\n        total = 0\n        for chunk in response.iter_content(chunk_size=_STREAM_CHUNK_SIZE):\n            if not chunk:\n                continue\n            total += len(chunk)\n            if total > max_bytes:\n                # Names the URL that served the body, which after a redirect is\n                # not the one that was requested.\n                raise ValueError(\n                    f\"Response body from '{response.url}' exceeds the \"\n                    f\"{max_bytes} byte limit.\"\n                )\n            chunks.append(chunk)\n\n        return (\n            b\"\".join(chunks),\n            response.headers.get(\"Content-Type\", \"\"),\n            response.url,\n        )\n    finally:\n        # Under stream=True each hop holds its connection until the body is read,\n        # so the redirects need closing too, not just the response we return.\n        for hop in response.history:\n            hop.close()\n        response.close()\n","sourceCodeStart":139,"sourceCodeEnd":174,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/security/safe_requests.py#L139-L174","documentation":"fetch_url_body() streams the response in _STREAM_CHUNK_SIZE chunks and accumulates the total; the moment the running total exceeds max_bytes it raises this ValueError naming response.url — which after redirects is the URL that actually served the body, not the one requested. This bounds memory usage and download time on untrusted URLs.","triggerScenarios":"Fetching a URL whose decoded body is larger than the max_bytes argument (e.g. a 50 MB PDF with max_bytes=10 MB); servers that ignore Range headers; content-encoding (gzip) inflating after download.","commonSituations":"Scraping tools pointed at pages that embed large media; default limits tuned for HTML hitting binary downloads; LLM agents fetching an arbitrary link found in text.","solutions":["Raise max_bytes for the specific call if the large body is expected and memory allows.","Check Content-Length first (when present) and skip or pre-emptively reject downloads over the cap.","Point the tool at a lighter endpoint (print/HTML version, API endpoint) instead of the full asset.","Do not retry with the same limit — the error is deterministic for that URL."],"exampleFix":"# before\nbody, ctype, final = fetch_url_body(url, max_bytes=1_000_000)\n\n# after\nbody, ctype, final = fetch_url_body(url, max_bytes=20_000_000)","handlingStrategy":"validation","validationCode":"def head_ok(url: str, max_bytes: int) -> bool:\n    import requests\n    h = requests.head(url, timeout=10, allow_redirects=True)\n    length = int(h.headers.get(\"Content-Length\", 0) or 0)\n    return length == 0 or length <= max_bytes","typeGuard":null,"tryCatchPattern":"try:\n    body, ctype, final = fetch_url_body(url, max_bytes=5_000_000)\nexcept ValueError as e:\n    if \"exceeds\" in str(e):\n        # deterministic for this URL; either raise the cap or skip\n        return None\n    raise","preventionTips":["Size the max_bytes argument to the content type you expect (HTML ~1-5 MB, PDFs much larger).","HEAD-check Content-Length first when fetching user/LLM-supplied URLs.","Don't retry with the same limit — the failure is content-deterministic."],"tags":["limits","http","streaming","memory"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}