crewAIInc/crewAI · error · ValueError

Error fetching content from URL {url}: {e!s}

Error message

Error fetching content from URL {url}: {e!s}

What it means

Raised by the shared fetch helpers in loaders/utils.py when an HTTP GET fails or returns a non-2xx status. The function performs safe_get(url, headers, timeout=30) then response.raise_for_status(); any exception from either step (DNS failure, timeout, 404/500) is re-raised as ValueError with the URL and original error text.

Source

Thrown at lib/crewai-tools/src/crewai_tools/rag/loaders/utils.py:41

    Raises:
        ValueError: If there's an error fetching the URL
    """
    from crewai_tools.security.safe_requests import safe_get

    headers = kwargs.get(
        "headers",
        {
            "Accept": accept_header,
            "User-Agent": f"Mozilla/5.0 (compatible; crewai-tools {loader_name})",
        },
    )

    try:
        response = safe_get(url, headers=headers, timeout=30)
        response.raise_for_status()
        return response.text
    except Exception as e:
        raise ValueError(f"Error fetching content from URL {url}: {e!s}") from e

View on GitHub (pinned to 754d7323be)

Solutions

1. Confirm the URL responds outside the app: curl -I -A 'Mozilla/5.0 (compatible; crewai-tools)' <url> to reproduce the status code.
2. If the server blocks the default User-Agent, pass custom headers via the loader's kwargs if supported, or fetch the content yourself and hand the text to the loader.
3. For slow servers, fetch the content with your own requests call using a longer timeout, then load the HTML string directly.
4. Catch ValueError and read the embedded original message to distinguish DNS/timeout from HTTP status failures.

Example fix

# before
result = web_loader.load(SourceContent(path="https://slow.example.com/huge"))

# after
import requests
resp = requests.get("https://slow.example.com/huge", timeout=120,
                    headers={"User-Agent": "my-bot/1.0"})
resp.raise_for_status()
# feed the retrieved text into your pipeline directly, skipping the util fetch

import requests

def url_is_fetchable(url: str, timeout: int = 10) -> bool:
    try:
        r = requests.head(url, timeout=timeout, allow_redirects=True,
                          headers={"User-Agent": "Mozilla/5.0 (compatible; crewai-tools)"})
        return r.status_code < 400
    except requests.RequestException:
        return False

try:
    text = fetch(url)
except ValueError as e:
    cause = str(e.__cause__ or "")
    if "404" in cause or "Not Found" in cause:
        mark_dead(url)
    else:
        retry_later(url)

When it happens

Trigger: Any loader that fetches a URL (webpage, sitemap, etc.) hitting: unreachable hosts, 30-second timeouts on slow servers, HTTP 4xx/5xx (raise_for_status), TLS certificate errors, or connections refused. The 30-second timeout is hardcoded, so slow endpoints reliably fail.

Common situations: Scraping sites that rate-limit or block the 'crewai-tools' User-Agent with 403; intranet URLs unreachable from the deployment environment; large pages that take over 30s to transfer; endpoints behind redirects to auth pages returning 401.

Related errors

• Error loading webpage {url}: {e!s}
• Failed to download template: {e}
• Unable to fetch documentation from {docs_url}: {e}
• Error fetching content from URL {url}: {e!s}
• Failed to download PDF from {url}: {e!s}

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/68bd4d0643e635c0. Report an issue: GitHub.