{"record":{"id":"53acf450afc310f3","repo":"crewAIInc/crewAI","slug":"unable-to-extract-content-from-documentation-site","errorCode":null,"errorMessage":"Unable to extract content from documentation site: {docs_url}","messagePattern":"Unable to extract content from documentation site: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/rag/loaders/docs_site_loader.py","lineNumber":63,"sourceCode":" for selector in [\n \"main\",\n \"article\",\n '[role=\"main\"]',\n \".content\",\n \"#content\",\n \".documentation\",\n ]:\n main_content = soup.select_one(selector)\n if main_content:\n break\n\n if not main_content:\n body = soup.find(\"body\")\n if isinstance(body, Tag):\n main_content = body\n\n if not main_content:\n raise ValueError(\n f\"Unable to extract content from documentation site: {docs_url}\"\n )\n\n text_parts = [f\"Title: {title_text}\", \"\"]\n\n headings = main_content.find_all([\"h1\", \"h2\", \"h3\"])\n if headings:\n text_parts.append(\"Table of Contents:\")\n for heading in headings[:15]:\n if not isinstance(heading, Tag):\n continue\n level = int(heading.name[1])\n indent = \" \" * (level - 1)\n text_parts.append(f\"{indent}- {heading.get_text(strip=True)}\")\n text_parts.append(\"\")\n\n text = main_content.get_text(separator=\"\\n\", strip=True)\n lines = [line.strip() for line in text.split(\"\\n\") if line.strip()]","sourceCodeStart":45,"sourceCodeEnd":81,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/rag/loaders/docs_site_loader.py#L45-L81","documentation":"Raised by DocsSiteLoader.load() when no main content region can be found in the fetched HTML. The loader tries a list of CSS selectors (article, .markdown-body, main, .documentation, ...); if none match it falls back to the tag. This error fires only when neither any selector nor a Tag exists, meaning the page is essentially empty, a JS-only shell, or non-HTML content.","triggerScenarios":"Loading a URL that returns an empty HTML document, a JSON/XML payload served without an HTML body, or a single-page app whose content is rendered client-side after fetch. Also possible for extremely minimal HTML fragments that lack a body element when parsed by html.parser.","commonSituations":"Docs sites that moved to client-side rendering (Next.js/Docusaurus shells with no SSR body); URLs that silently redirect to a login page or a blank interstitial; pointing the loader at an API endpoint instead of a docs page; anti-bot pages returning empty shells.","solutions":["Open the URL with view-source or curl and confirm the content is present in the server-rendered HTML; if not, the site needs a browser-based fetcher, not this loader.","Point the loader at a specific docs page that is server-rendered (many sites SSR real pages but not the index).","Check for redirects to auth walls and use a URL that serves public content.","If the content is JSON (e.g. an API), use a JSON-capable loader instead of DocsSiteLoader."],"exampleFix":"# before\nresult = DocsSiteLoader().load(SourceContent('https://app.example.com/docs')) # JS-only shell\n\n# after\n# use the SSR'd page or a stable versioned docs URL\nresult = DocsSiteLoader().load(SourceContent('https://docs.example.com/v2/getting-started'))","handlingStrategy":"fallback","validationCode":"import requests\\nfrom bs4 import BeautifulSoup\\n\\ndef has_rendered_body(url: str) -> bool:\\n html = requests.get(url, timeout=15).text\\n return BeautifulSoup(html, 'html.parser').find('body') is not None","typeGuard":null,"tryCatchPattern":"try:\\n result = DocsSiteLoader().load(source)\\nexcept ValueError as e:\\n if 'Unable to extract content' in str(e):\\n result = fallback_web_scraper(source.source)\\n else:\\n raise","preventionTips":["Prefer stable, versioned, server-rendered docs URLs.","Detect JS-only shells once and blacklist them for this loader.","Keep a browser-based fetcher as the fallback path for SPA docs."],"tags":["parsing","html","loader","rag"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}