crewAIInc/crewAI · error · ValueError

Unable to extract content from documentation site: {docs_url

Error message

Unable to extract content from documentation site: {docs_url}

What it means

Raised by DocsSiteLoader.load() when no main content region can be found in the fetched HTML. The loader tries a list of CSS selectors (article, .markdown-body, main, .documentation, ...); if none match it falls back to the <body> tag. This error fires only when neither any selector nor a <body> Tag exists, meaning the page is essentially empty, a JS-only shell, or non-HTML content.

Source

Thrown at lib/crewai-tools/src/crewai_tools/rag/loaders/docs_site_loader.py:63

        for selector in [
            "main",
            "article",
            '[role="main"]',
            ".content",
            "#content",
            ".documentation",
        ]:
            main_content = soup.select_one(selector)
            if main_content:
                break

        if not main_content:
            body = soup.find("body")
            if isinstance(body, Tag):
                main_content = body

        if not main_content:
            raise ValueError(
                f"Unable to extract content from documentation site: {docs_url}"
            )

        text_parts = [f"Title: {title_text}", ""]

        headings = main_content.find_all(["h1", "h2", "h3"])
        if headings:
            text_parts.append("Table of Contents:")
            for heading in headings[:15]:
                if not isinstance(heading, Tag):
                    continue
                level = int(heading.name[1])
                indent = "  " * (level - 1)
                text_parts.append(f"{indent}- {heading.get_text(strip=True)}")
            text_parts.append("")

        text = main_content.get_text(separator="\n", strip=True)
        lines = [line.strip() for line in text.split("\n") if line.strip()]

View on GitHub (pinned to 754d7323be)

Solutions

Open the URL with view-source or curl and confirm the content is present in the server-rendered HTML; if not, the site needs a browser-based fetcher, not this loader.
Point the loader at a specific docs page that is server-rendered (many sites SSR real pages but not the index).
Check for redirects to auth walls and use a URL that serves public content.
If the content is JSON (e.g. an API), use a JSON-capable loader instead of DocsSiteLoader.

Example fix

# before
result = DocsSiteLoader().load(SourceContent('https://app.example.com/docs'))  # JS-only shell

# after
# use the SSR'd page or a stable versioned docs URL
result = DocsSiteLoader().load(SourceContent('https://docs.example.com/v2/getting-started'))

Defensive patterns

Strategy: fallback

Validate before calling

import requests\nfrom bs4 import BeautifulSoup\n\ndef has_rendered_body(url: str) -> bool:\n    html = requests.get(url, timeout=15).text\n    return BeautifulSoup(html, 'html.parser').find('body') is not None

Try / catch

try:\n    result = DocsSiteLoader().load(source)\nexcept ValueError as e:\n    if 'Unable to extract content' in str(e):\n        result = fallback_web_scraper(source.source)\n    else:\n        raise

Prevention

Prefer stable, versioned, server-rendered docs URLs.
Detect JS-only shells once and blacklist them for this loader.
Keep a browser-based fetcher as the fallback path for SPA docs.

When it happens

Trigger: Loading a URL that returns an empty HTML document, a JSON/XML payload served without an HTML body, or a single-page app whose content is rendered client-side after fetch. Also possible for extremely minimal HTML fragments that lack a body element when parsed by html.parser.

Common situations: Docs sites that moved to client-side rendering (Next.js/Docusaurus shells with no SSR body); URLs that silently redirect to a login page or a blank interstitial; pointing the loader at an API endpoint instead of a docs page; anti-bot pages returning empty shells.

Related errors

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/53acf450afc310f3. Report an issue: GitHub.