{"record":{"id":"79c87f48b2a9940d","repo":"crewAIInc/crewAI","slug":"unable-to-fetch-documentation-from-docs-url-e","errorCode":null,"errorMessage":"Unable to fetch documentation from {docs_url}: {e}","messagePattern":"Unable to fetch documentation from (.+?): (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/rag/loaders/docs_site_loader.py","lineNumber":33,"sourceCode":"    \"\"\"Loader for documentation websites.\"\"\"\n\n    def load(self, source: SourceContent, **kwargs: Any) -> LoaderResult:  # type: ignore[override]\n        \"\"\"Load content from a documentation site.\n\n        Args:\n            source: Documentation site URL\n            **kwargs: Additional arguments\n\n        Returns:\n            LoaderResult with documentation content\n        \"\"\"\n        docs_url = source.source\n\n        try:\n            response = safe_get(docs_url, timeout=30)\n            response.raise_for_status()\n        except requests.RequestException as e:\n            raise ValueError(\n                f\"Unable to fetch documentation from {docs_url}: {e}\"\n            ) from e\n\n        soup = BeautifulSoup(response.text, \"html.parser\")\n\n        for script in soup([\"script\", \"style\"]):\n            script.decompose()\n\n        title = soup.find(\"title\")\n        title_text = title.get_text(strip=True) if title else \"Documentation\"\n\n        for selector in [\n            \"main\",\n            \"article\",\n            '[role=\"main\"]',\n            \".content\",\n            \"#content\",\n            \".documentation\",","sourceCodeStart":15,"sourceCodeEnd":51,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/rag/loaders/docs_site_loader.py#L15-L51","documentation":"Raised by DocsSiteLoader.load() when the HTTP request to the documentation URL fails. safe_get() plus raise_for_status() runs inside a try block; any requests.RequestException (DNS failure, connection refused, timeout after 30s, 404/500 status) is re-raised as ValueError with the URL and the underlying network error chained via 'from e'.","triggerScenarios":"Calling DocsSiteLoader().load(SourceContent('https://docs.example.com')) when the host is unreachable, the URL 404s, a proxy blocks the request, TLS verification fails, or the server takes longer than the hardcoded 30-second timeout.","commonSituations":"Corporate networks with egress proxies or SSL inspection that break requests to doc sites; offline development; typos in the docs URL; doc sites that block non-browser user agents with 403; slow sites that exceed the 30s timeout.","solutions":["Confirm the URL opens in a browser or with curl -I; fix typos or stale links to moved documentation.","If behind a proxy, set HTTPS_PROXY/HTTP_PROXY env vars or configure the session used by safe_get accordingly.","Retry once after a short delay — transient DNS/5xx failures are common; consider caching the fetched docs.","If the site blocks the client or is slow, pass a different mirror URL or raise the timeout by fetching the page yourself and handing the HTML to a parser."],"exampleFix":"# before\nresult = DocsSiteLoader().load(SourceContent('https://docs.exmaple.com/intro'))\n\n# after\nurl = 'https://docs.example.com/intro'  # fix typo\ntry:\n    result = DocsSiteLoader().load(SourceContent(url))\nexcept ValueError as e:\n    logger.warning('docs fetch failed, skipping: %s', e)\n    result = None","handlingStrategy":"retry","validationCode":"import requests\\n\\ndef docs_url_reachable(url: str) -> bool:\\n    try:\\n        return requests.head(url, timeout=10, allow_redirects=True).ok\\n    except requests.RequestException:\\n        return False","typeGuard":null,"tryCatchPattern":"for attempt in range(2):\\n    try:\\n        result = DocsSiteLoader().load(source)\\n        break\\n    except ValueError as e:\\n        if attempt == 1:\\n            logger.warning('docs fetch failed twice: %s', e)\\n            result = None","preventionTips":["Cache fetched docs pages so repeated runs never re-fetch.","Validate URLs with a HEAD request before queuing them.","Configure proxy env vars in restricted networks."],"tags":["network","http","loader","rag"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}