{"record":{"id":"fa48513107aa536c","repo":"crewAIInc/crewAI","slug":"error-loading-webpage-url-e-s","errorCode":null,"errorMessage":"Error loading webpage {url}: {e!s}","messagePattern":"Error loading webpage (.+?): (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/rag/loaders/webpage_loader.py","lineNumber":59,"sourceCode":"            title = (\n                soup.title.string.strip() if soup.title and soup.title.string else \"\"\n            )\n            metadata = {\n                \"url\": url,\n                \"title\": title,\n                \"status_code\": response.status_code,\n                \"content_type\": response.headers.get(\"content-type\", \"\"),\n            }\n\n            return LoaderResult(\n                content=text,\n                source=url,\n                metadata=metadata,\n                doc_id=self.generate_doc_id(source_ref=url, content=text),\n            )\n\n        except Exception as e:\n            raise ValueError(f\"Error loading webpage {url}: {e!s}\") from e\n","sourceCodeStart":41,"sourceCodeEnd":60,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/rag/loaders/webpage_loader.py#L41-L60","documentation":"The webpage loader's outer catch-all: any exception during the whole load flow — fetching, HTTP status checking, content-type inspection, or HTML-to-text extraction — is wrapped into ValueError('Error loading webpage {url}') with the original preserved as __cause__. It is the same failure surface as the utils fetch error (247) plus whatever the loader does after fetching.","triggerScenarios":"Calling WebPageLoader.load on a URL that 404s or 500s (raise_for_status inside), DNS/timeouts during fetch, or non-HTML content types that break the text extraction step. Also triggered by any parsing exception in the HTML cleaner on malformed markup.","commonSituations":"Crawling lists of URLs where some are dead or redirect to error pages; sites serving PDFs/binary at HTML URLs; SPAs returning empty shells that break extraction; corporate proxies returning 407.","solutions":["Test the URL with curl -L -o /dev/null -w '%{http_code} %{content_type}' <url> to see the effective status and type after redirects.","Inspect e.__cause__ in the catch block — it contains the real fetch or parse error, the outer message alone is generic.","Filter candidate URLs before loading (HEAD request, status < 400, content-type text/html).","For pages requiring JS rendering, fetch with a headless browser and pass the rendered HTML to your own text pipeline instead."],"exampleFix":"# before\ntry:\n    result = loader.load(SourceContent(path=url))\nexcept ValueError as e:\n    pass  # generic message, cause lost\n\n# after\ntry:\n    result = loader.load(SourceContent(path=url))\nexcept ValueError as e:\n    log.warning(\"webpage load failed url=%s cause=%r\", url, e.__cause__)","handlingStrategy":"try-catch","validationCode":"import requests\n\ndef is_loadable_webpage(url: str) -> bool:\n    try:\n        r = requests.get(url, timeout=15, stream=True,\n                         headers={\"User-Agent\": \"Mozilla/5.0\"})\n        ok = r.status_code < 400 and \"html\" in r.headers.get(\"content-type\", \"\")\n        r.close()\n        return ok\n    except requests.RequestException:\n        return False","typeGuard":null,"tryCatchPattern":"try:\n    result = web_loader.load(src)\nexcept ValueError as e:\n    log.warning(\"webpage failed url=%s cause=%r\", url, e.__cause__)\n    dead_urls.add(url)  # skip in future crawls","preventionTips":["Always log __cause__; the outer message is generic by design.","Crawl with per-URL error isolation so one bad page never stops the batch.","Skip non-HTML content types up front to avoid extraction-time failures."],"tags":["network","http","web-scraping","rag"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}