ScrapeGraphAI/Scrapegraph-ai · error · SearchRequestError

Serper search failed: {str(e)}

Error message

Serper search failed: {str(e)}

What it means

Raised by the Serper backend when the HTTP call or response parsing fails: network error, non-200 response, invalid JSON, or missing 'link' keys in results. Wrapped into SearchRequestError.

Source

Thrown at scrapegraphai/utils/research_web.py:416

            json=data,
            headers=headers,
            timeout=timeout,
        )
        response.raise_for_status()

        json_data = response.json()
        results = []

        # Extract organic search results
        for item in json_data.get("organic", []):
            if "link" in item:
                results.append(item["link"])
                if len(results) >= max_results:
                    break

        return results
    except Exception as e:
        raise SearchRequestError(f"Serper search failed: {str(e)}")


def format_proxy(proxy_config: Union[str, Dict, ProxyConfig]) -> str:
    """
    Format proxy configuration into a string.

    Args:
        proxy_config: Proxy configuration as string, dict, or ProxyConfig

    Returns:
        str: Formatted proxy string
    """
    if isinstance(proxy_config, str):
        return proxy_config

    if isinstance(proxy_config, dict):
        proxy_config = ProxyConfig(**proxy_config)

View on GitHub (pinned to 532dfffbf6)

Solutions

1. Check str(e) — 401/403 means bad key, 429 means quota
2. Validate the API key with a manual curl to google.serper.dev
3. Add retry with backoff for 5xx/transient failures
4. Reduce max_results if num is rejected

Example fix

// before
results = search_on_web(config)  # serper 401
// after
# verify key: curl -X POST google.serper.dev/search -H 'X-API-KEY: $KEY' ...
config.serper_api_key = os.getenv("SERPER_APIKEY")

import requests
def serper_key_ok(key):
    r = requests.post("https://google.serper.dev/search", headers={"X-API-KEY": key}, json={"q": "ping"}, timeout=10)
    return r.status_code == 200

for attempt in range(3):
    try:
        results = search_on_web(config); break
    except SearchRequestError as e:
        if attempt == 2: raise
        time.sleep(2 ** attempt)

When it happens

Trigger: Calling search_on_web with engine='serper' when serper.dev returns an error (invalid key, quota exceeded), the request times out at HTTP level, or the JSON payload lacks expected fields.

Common situations: Expired or invalid Serper API key (401/403); exhausted free quota; num exceeding API limits; transient serper.dev outages.

Related errors

• undetected_chromedriver is required for ChromiumLoader. Plea
• Search request timed out after {timeout} seconds
• Search request failed: {str(e)}
• Invalid search configuration: {str(e)}
• Bing search failed: {str(e)}

AI-assisted analysis of ScrapeGraphAI/Scrapegraph-ai@532dfffbf6 (2026-08-28). Data as JSON: /api/errors/579ff6fd692730b5. Report an issue: GitHub.