{"record":{"id":"68bd4d0643e635c0","repo":"crewAIInc/crewAI","slug":"error-fetching-content-from-url-url-e-s-68bd4d","errorCode":null,"errorMessage":"Error fetching content from URL {url}: {e!s}","messagePattern":"Error fetching content from URL (.+?): (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/rag/loaders/utils.py","lineNumber":41,"sourceCode":"    Raises:\n        ValueError: If there's an error fetching the URL\n    \"\"\"\n    from crewai_tools.security.safe_requests import safe_get\n\n    headers = kwargs.get(\n        \"headers\",\n        {\n            \"Accept\": accept_header,\n            \"User-Agent\": f\"Mozilla/5.0 (compatible; crewai-tools {loader_name})\",\n        },\n    )\n\n    try:\n        response = safe_get(url, headers=headers, timeout=30)\n        response.raise_for_status()\n        return response.text\n    except Exception as e:\n        raise ValueError(f\"Error fetching content from URL {url}: {e!s}\") from e\n","sourceCodeStart":23,"sourceCodeEnd":42,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/rag/loaders/utils.py#L23-L42","documentation":"Raised by the shared fetch helpers in loaders/utils.py when an HTTP GET fails or returns a non-2xx status. The function performs safe_get(url, headers, timeout=30) then response.raise_for_status(); any exception from either step (DNS failure, timeout, 404/500) is re-raised as ValueError with the URL and original error text.","triggerScenarios":"Any loader that fetches a URL (webpage, sitemap, etc.) hitting: unreachable hosts, 30-second timeouts on slow servers, HTTP 4xx/5xx (raise_for_status), TLS certificate errors, or connections refused. The 30-second timeout is hardcoded, so slow endpoints reliably fail.","commonSituations":"Scraping sites that rate-limit or block the 'crewai-tools' User-Agent with 403; intranet URLs unreachable from the deployment environment; large pages that take over 30s to transfer; endpoints behind redirects to auth pages returning 401.","solutions":["Confirm the URL responds outside the app: curl -I -A 'Mozilla/5.0 (compatible; crewai-tools)' <url> to reproduce the status code.","If the server blocks the default User-Agent, pass custom headers via the loader's kwargs if supported, or fetch the content yourself and hand the text to the loader.","For slow servers, fetch the content with your own requests call using a longer timeout, then load the HTML string directly.","Catch ValueError and read the embedded original message to distinguish DNS/timeout from HTTP status failures."],"exampleFix":"# before\nresult = web_loader.load(SourceContent(path=\"https://slow.example.com/huge\"))\n\n# after\nimport requests\nresp = requests.get(\"https://slow.example.com/huge\", timeout=120,\n                    headers={\"User-Agent\": \"my-bot/1.0\"})\nresp.raise_for_status()\n# feed the retrieved text into your pipeline directly, skipping the util fetch","handlingStrategy":"try-catch","validationCode":"import requests\n\ndef url_is_fetchable(url: str, timeout: int = 10) -> bool:\n    try:\n        r = requests.head(url, timeout=timeout, allow_redirects=True,\n                          headers={\"User-Agent\": \"Mozilla/5.0 (compatible; crewai-tools)\"})\n        return r.status_code < 400\n    except requests.RequestException:\n        return False","typeGuard":null,"tryCatchPattern":"try:\n    text = fetch(url)\nexcept ValueError as e:\n    cause = str(e.__cause__ or \"\")\n    if \"404\" in cause or \"Not Found\" in cause:\n        mark_dead(url)\n    else:\n        retry_later(url)","preventionTips":["Pre-filter URL lists with cheap HEAD requests before loading.","Assume a 30s hard timeout; fetch oversized/slow pages yourself with a longer timeout.","Cache successful fetches to avoid repeat network exposure."],"tags":["network","http","web-scraping","timeout"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}