crewAIInc/crewAI · error · ValueError

Unable to fetch documentation from {docs_url}: {e}

Error message

Unable to fetch documentation from {docs_url}: {e}

What it means

Raised by DocsSiteLoader.load() when the HTTP request to the documentation URL fails. safe_get() plus raise_for_status() runs inside a try block; any requests.RequestException (DNS failure, connection refused, timeout after 30s, 404/500 status) is re-raised as ValueError with the URL and the underlying network error chained via 'from e'.

Source

Thrown at lib/crewai-tools/src/crewai_tools/rag/loaders/docs_site_loader.py:33

    """Loader for documentation websites."""

    def load(self, source: SourceContent, **kwargs: Any) -> LoaderResult:  # type: ignore[override]
        """Load content from a documentation site.

        Args:
            source: Documentation site URL
            **kwargs: Additional arguments

        Returns:
            LoaderResult with documentation content
        """
        docs_url = source.source

        try:
            response = safe_get(docs_url, timeout=30)
            response.raise_for_status()
        except requests.RequestException as e:
            raise ValueError(
                f"Unable to fetch documentation from {docs_url}: {e}"
            ) from e

        soup = BeautifulSoup(response.text, "html.parser")

        for script in soup(["script", "style"]):
            script.decompose()

        title = soup.find("title")
        title_text = title.get_text(strip=True) if title else "Documentation"

        for selector in [
            "main",
            "article",
            '[role="main"]',
            ".content",
            "#content",
            ".documentation",

View on GitHub (pinned to 754d7323be)

Solutions

1. Confirm the URL opens in a browser or with curl -I; fix typos or stale links to moved documentation.
2. If behind a proxy, set HTTPS_PROXY/HTTP_PROXY env vars or configure the session used by safe_get accordingly.
3. Retry once after a short delay — transient DNS/5xx failures are common; consider caching the fetched docs.
4. If the site blocks the client or is slow, pass a different mirror URL or raise the timeout by fetching the page yourself and handing the HTML to a parser.

Example fix

# before
result = DocsSiteLoader().load(SourceContent('https://docs.exmaple.com/intro'))

# after
url = 'https://docs.example.com/intro'  # fix typo
try:
    result = DocsSiteLoader().load(SourceContent(url))
except ValueError as e:
    logger.warning('docs fetch failed, skipping: %s', e)
    result = None

import requests\n\ndef docs_url_reachable(url: str) -> bool:\n    try:\n        return requests.head(url, timeout=10, allow_redirects=True).ok\n    except requests.RequestException:\n        return False

for attempt in range(2):\n    try:\n        result = DocsSiteLoader().load(source)\n        break\n    except ValueError as e:\n        if attempt == 1:\n            logger.warning('docs fetch failed twice: %s', e)\n            result = None

When it happens

Trigger: Calling DocsSiteLoader().load(SourceContent('https://docs.example.com')) when the host is unreachable, the URL 404s, a proxy blocks the request, TLS verification fails, or the server takes longer than the hardcoded 30-second timeout.

Common situations: Corporate networks with egress proxies or SSL inspection that break requests to doc sites; offline development; typos in the docs URL; doc sites that block non-browser user agents with 403; slow sites that exceed the 30s timeout.

Related errors

• Error fetching content from URL {url}: {e!s}
• Failed to download PDF from {url}: {e!s}
• Error loading webpage {url}: {e!s}
• No loader defined for {self}
• Error loading loader for {self}: {e}

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/79c87f48b2a9940d. Report an issue: GitHub.