crewAIInc/crewAI · error · RuntimeError

Scraping failed: {e!s}

Error message

Scraping failed: {e!s}

What it means

Catch-all RuntimeError raised by ScrapegraphScrapeTool._run for any exception other than RateLimitError that occurs while calling _client.smartscraper(). The original exception is preserved as the cause (__cause__ via 'from e') and its text is embedded after 'Scraping failed: '. Typical underlying causes are network failures, auth errors (bad api_key), timeouts, or URL validation problems inside the client.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/scrapegraph_scrape_tool/scrapegraph_scrape_tool.py:185

        )

        if not website_url:
            raise ValueError("website_url is required")

        self._validate_url(website_url)

        try:
            if self._client is None:
                raise RuntimeError("Client not initialized")
            return self._client.smartscraper(
                website_url=website_url,
                user_prompt=user_prompt,
            )

        except RateLimitError:
            raise  # Re-raise rate limit errors
        except Exception as e:
            raise RuntimeError(f"Scraping failed: {e!s}") from e
        finally:
            # Always close the client
            if self._client is not None:
                self._client.close()

View on GitHub (pinned to 754d7323be)

Solutions

1. Inspect the exception chain (e.__cause__) or the text after 'Scraping failed:' to identify the real failure (auth vs network vs timeout).
2. Verify the API key: confirm SCRAPEGRAPH_API_KEY is set/valid and the tool's api_key argument is correct.
3. For network causes, check connectivity to the Scrapegraph endpoint (proxy, DNS, firewall) and retry with backoff.
4. Wrap calls in try/except RuntimeError and implement retry-with-backoff for transient causes.

Example fix

# before
try:
    out = tool.run(website_url=url)
except RuntimeError as e:
    raise  # loses the distinction between causes

# after
try:
    out = tool.run(website_url=url)
except RuntimeError as e:
    cause = e.__cause__
    if isinstance(cause, (ConnectionError, TimeoutError)):
        out = retry_with_backoff(lambda: tool.run(website_url=url))
    else:
        raise

import time

def safe_scrape(tool, url, retries=3):
    last = None
    for attempt in range(retries):
        try:
            return tool.run(website_url=url)
        except RuntimeError as e:
            cause = e.__cause__
            if isinstance(cause, (ConnectionError, TimeoutError)) and attempt < retries - 1:
                time.sleep(2 ** attempt)
                continue
            raise  # auth errors and final failures propagate
    raise last

When it happens

Trigger: smartscraper() raising requests.ConnectionError/Timeout (no network, DNS failure), 401/403 from an invalid SCRAPEGRAPH_API_KEY, or an SDK-level exception; _run catches it and re-raises RuntimeError(f'Scraping failed: {e}').

Common situations: Invalid or expired Scrapegraph API key; running in a sandbox/CI container without network egress to scrapegraph.ai; scraping a site that makes the upstream job time out; any transient network hiccup mid-request.

Related errors

• Skill {ref} not found. Ensure it has been published and you
• Error fetching triggers: {e}
• Failed to communicate with Agent Handler API: {e!s}
• Empty response from Scrapegraph API
• Rate limit exceeded: {error_msg}

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/b970f983cb91f42f. Report an issue: GitHub.