crewAIInc/crewAI · error · ValueError

Response body from '{response.url}' exceeds the {max_bytes}

Error message

Response body from '{response.url}' exceeds the {max_bytes} byte limit.

What it means

fetch_url_body() streams the response in _STREAM_CHUNK_SIZE chunks and accumulates the total; the moment the running total exceeds max_bytes it raises this ValueError naming response.url — which after redirects is the URL that actually served the body, not the one requested. This bounds memory usage and download time on untrusted URLs.

Source

Thrown at lib/crewai-tools/src/crewai_tools/security/safe_requests.py:157

        url,
        max_redirects=max_redirects,
        headers=headers,
        timeout=timeout,
        stream=True,
    )
    try:
        response.raise_for_status()

        chunks: list[bytes] = []
        total = 0
        for chunk in response.iter_content(chunk_size=_STREAM_CHUNK_SIZE):
            if not chunk:
                continue
            total += len(chunk)
            if total > max_bytes:
                # Names the URL that served the body, which after a redirect is
                # not the one that was requested.
                raise ValueError(
                    f"Response body from '{response.url}' exceeds the "
                    f"{max_bytes} byte limit."
                )
            chunks.append(chunk)

        return (
            b"".join(chunks),
            response.headers.get("Content-Type", ""),
            response.url,
        )
    finally:
        # Under stream=True each hop holds its connection until the body is read,
        # so the redirects need closing too, not just the response we return.
        for hop in response.history:
            hop.close()
        response.close()

View on GitHub (pinned to 754d7323be)

Solutions

Raise max_bytes for the specific call if the large body is expected and memory allows.
Check Content-Length first (when present) and skip or pre-emptively reject downloads over the cap.
Point the tool at a lighter endpoint (print/HTML version, API endpoint) instead of the full asset.
Do not retry with the same limit — the error is deterministic for that URL.

Example fix

# before
body, ctype, final = fetch_url_body(url, max_bytes=1_000_000)

# after
body, ctype, final = fetch_url_body(url, max_bytes=20_000_000)

Defensive patterns

Strategy: validation

Validate before calling

def head_ok(url: str, max_bytes: int) -> bool:
    import requests
    h = requests.head(url, timeout=10, allow_redirects=True)
    length = int(h.headers.get("Content-Length", 0) or 0)
    return length == 0 or length <= max_bytes

Try / catch

try:
    body, ctype, final = fetch_url_body(url, max_bytes=5_000_000)
except ValueError as e:
    if "exceeds" in str(e):
        # deterministic for this URL; either raise the cap or skip
        return None
    raise

Prevention

Size the max_bytes argument to the content type you expect (HTML ~1-5 MB, PDFs much larger).
HEAD-check Content-Length first when fetching user/LLM-supplied URLs.
Don't retry with the same limit — the failure is content-deterministic.

When it happens

Trigger: Fetching a URL whose decoded body is larger than the max_bytes argument (e.g. a 50 MB PDF with max_bytes=10 MB); servers that ignore Range headers; content-encoding (gzip) inflating after download.

Common situations: Scraping tools pointed at pages that embed large media; default limits tuned for HTML hitting binary downloads; LLM agents fetching an arbitrary link found in text.

Related errors

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/7949a5270fd37d80. Report an issue: GitHub.