{"record":{"id":"a881d2cbdb553dc1","repo":"crewAIInc/crewAI","slug":"empty-response-from-scrapegraph-api","errorCode":null,"errorMessage":"Empty response from Scrapegraph API","messagePattern":"Empty response from Scrapegraph API","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/tools/scrapegraph_scrape_tool/scrapegraph_scrape_tool.py","lineNumber":146,"sourceCode":"        if self.enable_logging:\n            sgai_logger.set_logging(level=\"INFO\")\n\n    @staticmethod\n    def _validate_url(url: str) -> None:\n        \"\"\"Validate URL format.\"\"\"\n        try:\n            result = urlparse(url)\n            if not all([result.scheme, result.netloc]):\n                raise ValueError\n        except Exception as e:\n            raise ValueError(\n                \"Invalid URL format. URL must include scheme (http/https) and domain\"\n            ) from e\n\n    def _handle_api_response(self, response: dict[str, Any]) -> str:\n        \"\"\"Handle and validate API response.\"\"\"\n        if not response:\n            raise RuntimeError(\"Empty response from Scrapegraph API\")\n\n        if \"error\" in response:\n            error_msg = response.get(\"error\", {}).get(\"message\", \"Unknown error\")\n            if \"rate limit\" in error_msg.lower():\n                raise RateLimitError(f\"Rate limit exceeded: {error_msg}\")\n            raise RuntimeError(f\"API error: {error_msg}\")\n\n        if \"result\" not in response:\n            raise RuntimeError(\"Invalid response format from Scrapegraph API\")\n\n        return str(response[\"result\"])\n\n    def _run(\n        self,\n        **kwargs: Any,\n    ) -> Any:\n        website_url = kwargs.get(\"website_url\", self.website_url)\n        user_prompt = (","sourceCodeStart":128,"sourceCodeEnd":164,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/tools/scrapegraph_scrape_tool/scrapegraph_scrape_tool.py#L128-L164","documentation":"Raised by ScrapegraphScrapeTool._handle_api_response when the API call succeeds transport-wise but returns a falsy response (empty dict or None). It is a RuntimeError (operation-level failure), distinct from the API-error and rate-limit branches that inspect response['error'], and from the missing-'result' format error.","triggerScenarios":"Calling the scrape operation and receiving {} or an empty body from the Scrapegraph API — e.g. an unexpected 200-with-empty-body, a client abstraction returning None on an edge case, or upstream API behavior changes returning empty payloads.","commonSituations":"Transient upstream issues; scraping targets the API cannot process and returns empty instead of an error; version mismatch between scrapegraph-py client expectations and live API responses.","solutions":["Retry the scrape once — empty responses are frequently transient","Verify the target website_url itself is reachable and not behind aggressive blocking that yields empty results","Upgrade scrapegraph-py (uv add 'scrapegraph-py@latest') in case the API contract changed","If reproducible for one URL only, that URL is likely unscrapeable — pick a different target or add site-specific options"],"exampleFix":"# before\nresult = tool.run(website_url='https://flaky-target.com')  # RuntimeError: Empty response\n\n# after\nfor attempt in range(3):\n    try:\n        result = tool.run(website_url='https://flaky-target.com')\n        break\n    except RuntimeError:\n        if attempt == 2:\n            raise\n","handlingStrategy":"retry","validationCode":"from urllib.parse import urlparse\n\ndef scrape_target_ok(u: str) -> bool:\n    p = urlparse(u)\n    return bool(p.scheme) and bool(p.netloc) and p.scheme in (\"http\", \"https\")\n\nassert scrape_target_ok(url), \"Target URL looks unscrapeable — check scheme and domain\"","typeGuard":null,"tryCatchPattern":"import time\nfor attempt in range(3):\n    try:\n        result = tool.run(website_url=url)\n        break\n    except RuntimeError as e:\n        if \"Empty response\" in str(e) and attempt < 2:\n            time.sleep(2 ** attempt)\n            continue\n        raise","preventionTips":["Retry empty responses once or twice — they are often transient","Smoke-test target URLs with a plain HTTP GET before handing them to the tool","Keep scrapegraph-py updated to track API response-format changes","Separate empty-response (RuntimeError) handling from RateLimitError and API-error branches"],"tags":["api","runtime","empty-response","scrapegraph","retry"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}