crewAIInc/crewAI · error · RateLimitError

Rate limit exceeded: {error_msg}

Error message

Rate limit exceeded: {error_msg}

What it means

Raised as RateLimitError by ScrapegraphScrapeTool._handle_api_response when the Scrapegraph API response contains an 'error' object whose message includes 'rate limit' (case-insensitive). It means your API key has exhausted its request quota or is sending requests too fast for the current plan. The message embeds the upstream error text from Scrapegraph so you can see whether it is a per-minute or per-month limit.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/scrapegraph_scrape_tool/scrapegraph_scrape_tool.py:151

        """Validate URL format."""
        try:
            result = urlparse(url)
            if not all([result.scheme, result.netloc]):
                raise ValueError
        except Exception as e:
            raise ValueError(
                "Invalid URL format. URL must include scheme (http/https) and domain"
            ) from e

    def _handle_api_response(self, response: dict[str, Any]) -> str:
        """Handle and validate API response."""
        if not response:
            raise RuntimeError("Empty response from Scrapegraph API")

        if "error" in response:
            error_msg = response.get("error", {}).get("message", "Unknown error")
            if "rate limit" in error_msg.lower():
                raise RateLimitError(f"Rate limit exceeded: {error_msg}")
            raise RuntimeError(f"API error: {error_msg}")

        if "result" not in response:
            raise RuntimeError("Invalid response format from Scrapegraph API")

        return str(response["result"])

    def _run(
        self,
        **kwargs: Any,
    ) -> Any:
        website_url = kwargs.get("website_url", self.website_url)
        user_prompt = (
            kwargs.get("user_prompt", self.user_prompt)
            or "Extract the main content of the webpage"
        )

        if not website_url:

View on GitHub (pinned to 754d7323be)

Solutions

1. Wait for the rate-limit window to reset (per-minute limits) or upgrade the Scrapegraph plan / add credits at scrapegraphai.com if it is a quota limit.
2. Add exponential backoff with retry around the tool call (the tool re-raises RateLimitError unchanged, so catch it specifically).
3. Throttle or serialize scraping calls (e.g. time.sleep between requests, a semaphore for concurrent crews).
4. Set SCRAPEGRAPH_API_KEY to a key on a plan that matches your request volume.

Example fix

// before
for url in urls:
    results.append(tool.run(url))  # bursts requests -> RateLimitError

// after
import time
for url in urls:
    try:
        results.append(tool.run(url))
    except RateLimitError:
        time.sleep(60)  # back off, then retry this url
        results.append(tool.run(url))

from crewai_tools.tools.scrapegraph_scrape_tool.scrapegraph_scrape_tool import RateLimitError
import time

def scrape_with_backoff(tool, url, max_retries=5):
    for attempt in range(max_retries):
        try:
            return tool.run(website_url=url)
        except RateLimitError:
            wait = 2 ** attempt * 30  # 30s, 60s, 120s...
            time.sleep(wait)
    raise RuntimeError(f"still rate-limited after {max_retries} attempts")

When it happens

Trigger: Calling ScrapegraphScrapeTool._run (or the tool from an agent) such that _client.smartscraper() returns a dict with key 'error' whose 'message' contains 'rate limit'; e.g. bursting many scrapes in a loop, or exceeding the free tier's monthly credits.

Common situations: Batch-scraping dozens of URLs in quick succession on a free/hobby Scrapegraph plan; sharing one SCRAPEGRAPH_API_KEY across multiple concurrent crews or processes; a long-running job that eventually crosses the plan quota mid-run.

Related errors

• Empty response from Scrapegraph API
• Invalid response format from Scrapegraph API
• Scraping failed: {e!s}
• Project name '{name}' would generate invalid Python class na
• Failed to retrieve organization list: {e!s}

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/d09aa633ba682b47. Report an issue: GitHub.