{"record":{"id":"b970f983cb91f42f","repo":"crewAIInc/crewAI","slug":"scraping-failed-e-s","errorCode":null,"errorMessage":"Scraping failed: {e!s}","messagePattern":"Scraping failed: (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/tools/scrapegraph_scrape_tool/scrapegraph_scrape_tool.py","lineNumber":185,"sourceCode":"        )\n\n        if not website_url:\n            raise ValueError(\"website_url is required\")\n\n        self._validate_url(website_url)\n\n        try:\n            if self._client is None:\n                raise RuntimeError(\"Client not initialized\")\n            return self._client.smartscraper(\n                website_url=website_url,\n                user_prompt=user_prompt,\n            )\n\n        except RateLimitError:\n            raise  # Re-raise rate limit errors\n        except Exception as e:\n            raise RuntimeError(f\"Scraping failed: {e!s}\") from e\n        finally:\n            # Always close the client\n            if self._client is not None:\n                self._client.close()\n","sourceCodeStart":167,"sourceCodeEnd":190,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/tools/scrapegraph_scrape_tool/scrapegraph_scrape_tool.py#L167-L190","documentation":"Catch-all RuntimeError raised by ScrapegraphScrapeTool._run for any exception other than RateLimitError that occurs while calling _client.smartscraper(). The original exception is preserved as the cause (__cause__ via 'from e') and its text is embedded after 'Scraping failed: '. Typical underlying causes are network failures, auth errors (bad api_key), timeouts, or URL validation problems inside the client.","triggerScenarios":"smartscraper() raising requests.ConnectionError/Timeout (no network, DNS failure), 401/403 from an invalid SCRAPEGRAPH_API_KEY, or an SDK-level exception; _run catches it and re-raises RuntimeError(f'Scraping failed: {e}').","commonSituations":"Invalid or expired Scrapegraph API key; running in a sandbox/CI container without network egress to scrapegraph.ai; scraping a site that makes the upstream job time out; any transient network hiccup mid-request.","solutions":["Inspect the exception chain (e.__cause__) or the text after 'Scraping failed:' to identify the real failure (auth vs network vs timeout).","Verify the API key: confirm SCRAPEGRAPH_API_KEY is set/valid and the tool's api_key argument is correct.","For network causes, check connectivity to the Scrapegraph endpoint (proxy, DNS, firewall) and retry with backoff.","Wrap calls in try/except RuntimeError and implement retry-with-backoff for transient causes."],"exampleFix":"# before\ntry:\n    out = tool.run(website_url=url)\nexcept RuntimeError as e:\n    raise  # loses the distinction between causes\n\n# after\ntry:\n    out = tool.run(website_url=url)\nexcept RuntimeError as e:\n    cause = e.__cause__\n    if isinstance(cause, (ConnectionError, TimeoutError)):\n        out = retry_with_backoff(lambda: tool.run(website_url=url))\n    else:\n        raise","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"import time\n\ndef safe_scrape(tool, url, retries=3):\n    last = None\n    for attempt in range(retries):\n        try:\n            return tool.run(website_url=url)\n        except RuntimeError as e:\n            cause = e.__cause__\n            if isinstance(cause, (ConnectionError, TimeoutError)) and attempt < retries - 1:\n                time.sleep(2 ** attempt)\n                continue\n            raise  # auth errors and final failures propagate\n    raise last","preventionTips":["Inspect e.__cause__ to distinguish auth (fix key) from network (retry) failures.","Verify the API key and network egress to scrapegraph.ai before batch runs.","Catch RuntimeError and RateLimitError separately; only the latter should blindly retry."],"tags":["wrapper-exception","network","scrapegraph","auth","retry"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}