ScrapeGraphAI/Scrapegraph-ai · error · RuntimeError

scrapegraph-py request failed

Error message

scrapegraph-py request failed

What it means

RuntimeError raised by _unwrap_result when a scrapegraph-py SDK response object has both 'status' and 'data' attributes but status != 'success'. The message defaults to 'scrapegraph-py request failed' unless the result carries an 'error' attribute, which then replaces the default text.

Source

Thrown at scrapegraphai/integrations/scrapegraph_py_compat.py:45

        raise ImportError(
            "scrapegraph_py is not installed. Install it with 'pip install scrapegraph-py'."
        ) from e


def _schema_to_dict(schema: Optional[Type[BaseModel]]) -> Optional[dict]:
    if schema is None:
        return None
    if isinstance(schema, dict):
        return schema
    if isinstance(schema, type) and issubclass(schema, BaseModel):
        return schema.model_json_schema()
    return None


def _unwrap_result(result: Any) -> dict:
    if hasattr(result, "status") and hasattr(result, "data"):
        if result.status != "success":
            raise RuntimeError(
                getattr(result, "error", "scrapegraph-py request failed")
            )
        data = result.data
        if hasattr(data, "model_dump"):
            return data.model_dump(by_alias=True, exclude_none=True)
        return data if isinstance(data, dict) else {"data": data}
    return result


def extract(
    api_key: Optional[str],
    url: str,
    prompt: str,
    schema: Optional[Type[BaseModel]] = None,
) -> dict:
    """Call the scrapegraph-py extract endpoint across SDK versions."""
    api = _detect_api()

View on GitHub (pinned to 532dfffbf6)

Solutions

  1. If the raised message is generic, inspect the SDK response object / enable logging to get the underlying error detail.
  2. Verify your scrapegraph_py API key and remaining credits in the ScrapeGraph Cloud dashboard.
  3. Validate the URL and schema before sending; ensure the schema is a dict or serializable BaseModel.
  4. Retry once with backoff for transient server-side failures.
Defensive patterns

Strategy: retry

Validate before calling

def validate_request(url: str, schema=None) -> None:
    from urllib.parse import urlparse
    assert urlparse(url).scheme in ('http', 'https'), f'invalid url: {url}'
    if schema is not None:
        assert hasattr(schema, 'model_json_schema') or isinstance(schema, dict), 'schema must be a BaseModel or dict'

Try / catch

for attempt in range(3):
    try:
        return extract(url, schema=schema)
    except RuntimeError as e:
        msg = str(e)
        if 'request failed' in msg and attempt < 2:
            time.sleep(2 ** attempt)
            continue
        raise SystemExit(f'scrapegraph-py API error: {msg}') from e

Prevention

When it happens

Trigger: Any extract/scrape/search call through the compat layer whose API response has status 'error' (or anything non-'success') without an error field; triggered from _unwrap_result used by all three public helpers.

Common situations: Invalid or expired SGAI API key; request credits exhausted; malformed request payload (bad URL, invalid schema serialization); transient API-side failures.

Related errors

AI-assisted analysis of ScrapeGraphAI/Scrapegraph-ai@532dfffbf6 (2026-08-28). Data as JSON: /api/errors/5ef3dc0cfed807df. Report an issue: GitHub.