crewAIInc/crewAI · error · RuntimeError

Empty response from Scrapegraph API

Error message

Empty response from Scrapegraph API

What it means

Raised by ScrapegraphScrapeTool._handle_api_response when the API call succeeds transport-wise but returns a falsy response (empty dict or None). It is a RuntimeError (operation-level failure), distinct from the API-error and rate-limit branches that inspect response['error'], and from the missing-'result' format error.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/scrapegraph_scrape_tool/scrapegraph_scrape_tool.py:146

        if self.enable_logging:
            sgai_logger.set_logging(level="INFO")

    @staticmethod
    def _validate_url(url: str) -> None:
        """Validate URL format."""
        try:
            result = urlparse(url)
            if not all([result.scheme, result.netloc]):
                raise ValueError
        except Exception as e:
            raise ValueError(
                "Invalid URL format. URL must include scheme (http/https) and domain"
            ) from e

    def _handle_api_response(self, response: dict[str, Any]) -> str:
        """Handle and validate API response."""
        if not response:
            raise RuntimeError("Empty response from Scrapegraph API")

        if "error" in response:
            error_msg = response.get("error", {}).get("message", "Unknown error")
            if "rate limit" in error_msg.lower():
                raise RateLimitError(f"Rate limit exceeded: {error_msg}")
            raise RuntimeError(f"API error: {error_msg}")

        if "result" not in response:
            raise RuntimeError("Invalid response format from Scrapegraph API")

        return str(response["result"])

    def _run(
        self,
        **kwargs: Any,
    ) -> Any:
        website_url = kwargs.get("website_url", self.website_url)
        user_prompt = (

View on GitHub (pinned to 754d7323be)

Solutions

1. Retry the scrape once — empty responses are frequently transient
2. Verify the target website_url itself is reachable and not behind aggressive blocking that yields empty results
3. Upgrade scrapegraph-py (uv add 'scrapegraph-py@latest') in case the API contract changed
4. If reproducible for one URL only, that URL is likely unscrapeable — pick a different target or add site-specific options

Example fix

# before
result = tool.run(website_url='https://flaky-target.com')  # RuntimeError: Empty response

# after
for attempt in range(3):
    try:
        result = tool.run(website_url='https://flaky-target.com')
        break
    except RuntimeError:
        if attempt == 2:
            raise

from urllib.parse import urlparse

def scrape_target_ok(u: str) -> bool:
    p = urlparse(u)
    return bool(p.scheme) and bool(p.netloc) and p.scheme in ("http", "https")

assert scrape_target_ok(url), "Target URL looks unscrapeable — check scheme and domain"

import time
for attempt in range(3):
    try:
        result = tool.run(website_url=url)
        break
    except RuntimeError as e:
        if "Empty response" in str(e) and attempt < 2:
            time.sleep(2 ** attempt)
            continue
        raise

When it happens

Trigger: Calling the scrape operation and receiving {} or an empty body from the Scrapegraph API — e.g. an unexpected 200-with-empty-body, a client abstraction returning None on an edge case, or upstream API behavior changes returning empty payloads.

Common situations: Transient upstream issues; scraping targets the API cannot process and returns empty instead of an error; version mismatch between scrapegraph-py client expectations and live API responses.

Related errors

• Rate limit exceeded: {error_msg}
• Scraping failed: {e!s}
• Empty response from Serper API
• Project name '{name}' would generate invalid Python class na
• Failed to retrieve organization list: {e!s}

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/a881d2cbdb553dc1. Report an issue: GitHub.