{"record":{"id":"d09aa633ba682b47","repo":"crewAIInc/crewAI","slug":"rate-limit-exceeded-error-msg","errorCode":null,"errorMessage":"Rate limit exceeded: {error_msg}","messagePattern":"Rate limit exceeded: (.+?)","errorType":"exception","errorClass":"RateLimitError","httpStatus":429,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/tools/scrapegraph_scrape_tool/scrapegraph_scrape_tool.py","lineNumber":151,"sourceCode":"        \"\"\"Validate URL format.\"\"\"\n        try:\n            result = urlparse(url)\n            if not all([result.scheme, result.netloc]):\n                raise ValueError\n        except Exception as e:\n            raise ValueError(\n                \"Invalid URL format. URL must include scheme (http/https) and domain\"\n            ) from e\n\n    def _handle_api_response(self, response: dict[str, Any]) -> str:\n        \"\"\"Handle and validate API response.\"\"\"\n        if not response:\n            raise RuntimeError(\"Empty response from Scrapegraph API\")\n\n        if \"error\" in response:\n            error_msg = response.get(\"error\", {}).get(\"message\", \"Unknown error\")\n            if \"rate limit\" in error_msg.lower():\n                raise RateLimitError(f\"Rate limit exceeded: {error_msg}\")\n            raise RuntimeError(f\"API error: {error_msg}\")\n\n        if \"result\" not in response:\n            raise RuntimeError(\"Invalid response format from Scrapegraph API\")\n\n        return str(response[\"result\"])\n\n    def _run(\n        self,\n        **kwargs: Any,\n    ) -> Any:\n        website_url = kwargs.get(\"website_url\", self.website_url)\n        user_prompt = (\n            kwargs.get(\"user_prompt\", self.user_prompt)\n            or \"Extract the main content of the webpage\"\n        )\n\n        if not website_url:","sourceCodeStart":133,"sourceCodeEnd":169,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/tools/scrapegraph_scrape_tool/scrapegraph_scrape_tool.py#L133-L169","documentation":"Raised as RateLimitError by ScrapegraphScrapeTool._handle_api_response when the Scrapegraph API response contains an 'error' object whose message includes 'rate limit' (case-insensitive). It means your API key has exhausted its request quota or is sending requests too fast for the current plan. The message embeds the upstream error text from Scrapegraph so you can see whether it is a per-minute or per-month limit.","triggerScenarios":"Calling ScrapegraphScrapeTool._run (or the tool from an agent) such that _client.smartscraper() returns a dict with key 'error' whose 'message' contains 'rate limit'; e.g. bursting many scrapes in a loop, or exceeding the free tier's monthly credits.","commonSituations":"Batch-scraping dozens of URLs in quick succession on a free/hobby Scrapegraph plan; sharing one SCRAPEGRAPH_API_KEY across multiple concurrent crews or processes; a long-running job that eventually crosses the plan quota mid-run.","solutions":["Wait for the rate-limit window to reset (per-minute limits) or upgrade the Scrapegraph plan / add credits at scrapegraphai.com if it is a quota limit.","Add exponential backoff with retry around the tool call (the tool re-raises RateLimitError unchanged, so catch it specifically).","Throttle or serialize scraping calls (e.g. time.sleep between requests, a semaphore for concurrent crews).","Set SCRAPEGRAPH_API_KEY to a key on a plan that matches your request volume."],"exampleFix":"// before\nfor url in urls:\n    results.append(tool.run(url))  # bursts requests -> RateLimitError\n\n// after\nimport time\nfor url in urls:\n    try:\n        results.append(tool.run(url))\n    except RateLimitError:\n        time.sleep(60)  # back off, then retry this url\n        results.append(tool.run(url))","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"from crewai_tools.tools.scrapegraph_scrape_tool.scrapegraph_scrape_tool import RateLimitError\nimport time\n\ndef scrape_with_backoff(tool, url, max_retries=5):\n    for attempt in range(max_retries):\n        try:\n            return tool.run(website_url=url)\n        except RateLimitError:\n            wait = 2 ** attempt * 30  # 30s, 60s, 120s...\n            time.sleep(wait)\n    raise RuntimeError(f\"still rate-limited after {max_retries} attempts\")","preventionTips":["Throttle: keep at least ~1s between Scrapegraph calls and serialize concurrent crews sharing one key.","Match your Scrapegraph plan size to expected request volume before running batch jobs.","Alert on RateLimitError separately from other failures so quota issues surface immediately."],"tags":["rate-limit","scraping","api","scrapegraph","retry"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}