crewAIInc/crewAI · error · RuntimeError

Invalid response format from Scrapegraph API

Error message

Invalid response format from Scrapegraph API

What it means

Raised as RuntimeError by ScrapegraphScrapeTool._handle_api_response when the Scrapegraph API returned a non-empty dict that has neither an 'error' key nor a 'result' key. The tool only accepts responses shaped {'result': ...} (or {'error': ...}), so any other schema — an unexpected status payload, a changed API version, or a proxy/gateway response — is treated as invalid. This indicates a contract mismatch between the SDK client and the live API rather than a problem with your inputs.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/scrapegraph_scrape_tool/scrapegraph_scrape_tool.py:155

                raise ValueError
        except Exception as e:
            raise ValueError(
                "Invalid URL format. URL must include scheme (http/https) and domain"
            ) from e

    def _handle_api_response(self, response: dict[str, Any]) -> str:
        """Handle and validate API response."""
        if not response:
            raise RuntimeError("Empty response from Scrapegraph API")

        if "error" in response:
            error_msg = response.get("error", {}).get("message", "Unknown error")
            if "rate limit" in error_msg.lower():
                raise RateLimitError(f"Rate limit exceeded: {error_msg}")
            raise RuntimeError(f"API error: {error_msg}")

        if "result" not in response:
            raise RuntimeError("Invalid response format from Scrapegraph API")

        return str(response["result"])

    def _run(
        self,
        **kwargs: Any,
    ) -> Any:
        website_url = kwargs.get("website_url", self.website_url)
        user_prompt = (
            kwargs.get("user_prompt", self.user_prompt)
            or "Extract the main content of the webpage"
        )

        if not website_url:
            raise ValueError("website_url is required")

        self._validate_url(website_url)

View on GitHub (pinned to 754d7323be)

Solutions

1. Upgrade the scrapegraph SDK dependency so the client matches the current API response format (uv add / uv upgrade scrapegraph-sdk in lib/crewai-tools).
2. Log the raw response dict (monkeypatch or debug the client call) to see what keys actually came back before deciding.
3. Check the Scrapegraph status page and changelog for response-format changes or partial outages that return non-standard payloads.
4. If a proxy is intercepting, bypass it or add the API host to NO_PROXY.

Example fix

# before
result = tool.run(website_url="https://example.com")  # RuntimeError: Invalid response format

# after: inspect the raw envelope first
resp = client.smartscraper(website_url=url, user_prompt=prompt)
if "result" not in resp:
    logger.error("unexpected envelope: %s", resp.keys())
result = resp.get("result") or resp.get("data")

def is_valid_envelope(response: dict) -> bool:
    return isinstance(response, dict) and ("result" in response or "error" in response)

def is_scrapegraph_envelope(v) -> bool:
    """True when v matches {'result': ...} or {'error': {'message': ...}}."""
    return (
        isinstance(v, dict)
        and ("result" in v or ("error" in v and isinstance(v["error"], dict)))
    )

try:
    out = tool.run(website_url=url)
except RuntimeError as e:
    if "Invalid response format" in str(e):
        logger.error("scrapegraph envelope changed — inspect raw response; may need SDK upgrade")
        raise
    raise

When it happens

Trigger: smartscraper() returns e.g. {'data': ...}, {'message': 'ok'} or a gateway JSON body lacking both 'error' and 'result'; typically after Scrapegraph changes its response envelope or when the wrong endpoint/base URL is hit.

Common situations: Scrapegraph SDK version pinned in the project lags behind a live API change; a corporate proxy returns its own JSON instead of the API's; using a staging or EU endpoint with a different response shape.

Related errors

• Rate limit exceeded: {error_msg}
• Job failed: {status_data}
• Failed to install oxylabs package
• `oxylabs` package not found, please run `uv add oxylabs`
• Failed to install oxylabs package

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/3cec69f11aecf60a. Report an issue: GitHub.