{"record":{"id":"3cec69f11aecf60a","repo":"crewAIInc/crewAI","slug":"invalid-response-format-from-scrapegraph-api","errorCode":null,"errorMessage":"Invalid response format from Scrapegraph API","messagePattern":"Invalid response format from Scrapegraph API","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/tools/scrapegraph_scrape_tool/scrapegraph_scrape_tool.py","lineNumber":155,"sourceCode":"                raise ValueError\n        except Exception as e:\n            raise ValueError(\n                \"Invalid URL format. URL must include scheme (http/https) and domain\"\n            ) from e\n\n    def _handle_api_response(self, response: dict[str, Any]) -> str:\n        \"\"\"Handle and validate API response.\"\"\"\n        if not response:\n            raise RuntimeError(\"Empty response from Scrapegraph API\")\n\n        if \"error\" in response:\n            error_msg = response.get(\"error\", {}).get(\"message\", \"Unknown error\")\n            if \"rate limit\" in error_msg.lower():\n                raise RateLimitError(f\"Rate limit exceeded: {error_msg}\")\n            raise RuntimeError(f\"API error: {error_msg}\")\n\n        if \"result\" not in response:\n            raise RuntimeError(\"Invalid response format from Scrapegraph API\")\n\n        return str(response[\"result\"])\n\n    def _run(\n        self,\n        **kwargs: Any,\n    ) -> Any:\n        website_url = kwargs.get(\"website_url\", self.website_url)\n        user_prompt = (\n            kwargs.get(\"user_prompt\", self.user_prompt)\n            or \"Extract the main content of the webpage\"\n        )\n\n        if not website_url:\n            raise ValueError(\"website_url is required\")\n\n        self._validate_url(website_url)\n","sourceCodeStart":137,"sourceCodeEnd":173,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/tools/scrapegraph_scrape_tool/scrapegraph_scrape_tool.py#L137-L173","documentation":"Raised as RuntimeError by ScrapegraphScrapeTool._handle_api_response when the Scrapegraph API returned a non-empty dict that has neither an 'error' key nor a 'result' key. The tool only accepts responses shaped {'result': ...} (or {'error': ...}), so any other schema — an unexpected status payload, a changed API version, or a proxy/gateway response — is treated as invalid. This indicates a contract mismatch between the SDK client and the live API rather than a problem with your inputs.","triggerScenarios":"smartscraper() returns e.g. {'data': ...}, {'message': 'ok'} or a gateway JSON body lacking both 'error' and 'result'; typically after Scrapegraph changes its response envelope or when the wrong endpoint/base URL is hit.","commonSituations":"Scrapegraph SDK version pinned in the project lags behind a live API change; a corporate proxy returns its own JSON instead of the API's; using a staging or EU endpoint with a different response shape.","solutions":["Upgrade the scrapegraph SDK dependency so the client matches the current API response format (uv add / uv upgrade scrapegraph-sdk in lib/crewai-tools).","Log the raw response dict (monkeypatch or debug the client call) to see what keys actually came back before deciding.","Check the Scrapegraph status page and changelog for response-format changes or partial outages that return non-standard payloads.","If a proxy is intercepting, bypass it or add the API host to NO_PROXY."],"exampleFix":"# before\nresult = tool.run(website_url=\"https://example.com\")  # RuntimeError: Invalid response format\n\n# after: inspect the raw envelope first\nresp = client.smartscraper(website_url=url, user_prompt=prompt)\nif \"result\" not in resp:\n    logger.error(\"unexpected envelope: %s\", resp.keys())\nresult = resp.get(\"result\") or resp.get(\"data\")","handlingStrategy":"type-guard","validationCode":"def is_valid_envelope(response: dict) -> bool:\n    return isinstance(response, dict) and (\"result\" in response or \"error\" in response)","typeGuard":"def is_scrapegraph_envelope(v) -> bool:\n    \"\"\"True when v matches {'result': ...} or {'error': {'message': ...}}.\"\"\"\n    return (\n        isinstance(v, dict)\n        and (\"result\" in v or (\"error\" in v and isinstance(v[\"error\"], dict)))\n    )","tryCatchPattern":"try:\n    out = tool.run(website_url=url)\nexcept RuntimeError as e:\n    if \"Invalid response format\" in str(e):\n        logger.error(\"scrapegraph envelope changed — inspect raw response; may need SDK upgrade\")\n        raise\n    raise","preventionTips":["Pin and regularly upgrade scrapegraph-sdk so client parsing matches the live API.","Log raw API responses in debug mode to detect envelope drift early.","Treat this error as a version-contract issue, not a data issue — don't retry the same call unchanged."],"tags":["api-contract","scraping","scrapegraph","response-format"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}