crewAIInc/crewAI · error · ValueError

Error loading webpage {url}: {e!s}

Error message

Error loading webpage {url}: {e!s}

What it means

The webpage loader's outer catch-all: any exception during the whole load flow — fetching, HTTP status checking, content-type inspection, or HTML-to-text extraction — is wrapped into ValueError('Error loading webpage {url}') with the original preserved as __cause__. It is the same failure surface as the utils fetch error (247) plus whatever the loader does after fetching.

Source

Thrown at lib/crewai-tools/src/crewai_tools/rag/loaders/webpage_loader.py:59

            title = (
                soup.title.string.strip() if soup.title and soup.title.string else ""
            )
            metadata = {
                "url": url,
                "title": title,
                "status_code": response.status_code,
                "content_type": response.headers.get("content-type", ""),
            }

            return LoaderResult(
                content=text,
                source=url,
                metadata=metadata,
                doc_id=self.generate_doc_id(source_ref=url, content=text),
            )

        except Exception as e:
            raise ValueError(f"Error loading webpage {url}: {e!s}") from e

View on GitHub (pinned to 754d7323be)

Solutions

1. Test the URL with curl -L -o /dev/null -w '%{http_code} %{content_type}' <url> to see the effective status and type after redirects.
2. Inspect e.__cause__ in the catch block — it contains the real fetch or parse error, the outer message alone is generic.
3. Filter candidate URLs before loading (HEAD request, status < 400, content-type text/html).
4. For pages requiring JS rendering, fetch with a headless browser and pass the rendered HTML to your own text pipeline instead.

Example fix

# before
try:
    result = loader.load(SourceContent(path=url))
except ValueError as e:
    pass  # generic message, cause lost

# after
try:
    result = loader.load(SourceContent(path=url))
except ValueError as e:
    log.warning("webpage load failed url=%s cause=%r", url, e.__cause__)

import requests

def is_loadable_webpage(url: str) -> bool:
    try:
        r = requests.get(url, timeout=15, stream=True,
                         headers={"User-Agent": "Mozilla/5.0"})
        ok = r.status_code < 400 and "html" in r.headers.get("content-type", "")
        r.close()
        return ok
    except requests.RequestException:
        return False

try:
    result = web_loader.load(src)
except ValueError as e:
    log.warning("webpage failed url=%s cause=%r", url, e.__cause__)
    dead_urls.add(url)  # skip in future crawls

When it happens

Trigger: Calling WebPageLoader.load on a URL that 404s or 500s (raise_for_status inside), DNS/timeouts during fetch, or non-HTML content types that break the text extraction step. Also triggered by any parsing exception in the HTML cleaner on malformed markup.

Common situations: Crawling lists of URLs where some are dead or redirect to error pages; sites serving PDFs/binary at HTML URLs; SPAs returning empty shells that break extraction; corporate proxies returning 407.

Related errors

• Unable to fetch documentation from {docs_url}: {e}
• Error fetching content from URL {url}: {e!s}
• Error fetching content from URL {url}: {e!s}
• Failed to download PDF from {url}: {e!s}
• Too many redirects while fetching URL: {url}

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/fa48513107aa536c. Report an issue: GitHub.