ScrapeGraphAI/Scrapegraph-ai · error · SearchRequestError

Search request failed: {str(e)}

Error message

Search request failed: {str(e)}

What it means

Raised by search_on_web when a search request fails with a non-timeout requests exception (connection error, HTTP error, malformed URL, TLS failure). It wraps the original requests.RequestException message into SearchRequestError for a uniform error surface.

Source

Thrown at scrapegraphai/utils/research_web.py:241

                config.query, config.max_results, config.timeout, formatted_proxy
            )

        elif config.search_engine == "searxng":
            results = _search_searxng(
                config.query, config.max_results, config.port, config.timeout
            )

        elif config.search_engine == "serper":
            results = _search_serper(
                config.query, config.max_results, config.serper_api_key, config.timeout
            )

        return filter_pdf_links(results)

    except requests.Timeout:
        raise TimeoutError(f"Search request timed out after {timeout} seconds")
    except requests.RequestException as e:
        raise SearchRequestError(f"Search request failed: {str(e)}")
    except ValueError as e:
        raise SearchConfigError(f"Invalid search configuration: {str(e)}")


def _search_duckduckgo(
    query: str, max_results: int, proxy: Optional[str] = None
) -> List[str]:
    """
    Helper function for DuckDuckGo search using the ``ddgs`` package.

    The ``duckduckgo-search`` package was renamed to ``ddgs``; recent
    ``langchain-community`` releases import ``from ddgs import DDGS``, which
    silently broke the previous langchain-based implementation. This calls
    ``ddgs`` directly so results no longer depend on parsing a formatted string.

    Args:
        query (str): Search query
        max_results (int): Maximum number of results to return

View on GitHub (pinned to 532dfffbf6)

Solutions

  1. Inspect the wrapped message (str(e)) to identify the HTTP/connection cause
  2. Add or fix proxy configuration in the SearchConfig
  3. If 429/403, reduce request frequency or switch to an API-based engine (serper)
  4. For SSL issues, verify certificates or configure the environment's CA bundle

Example fix

// before
results = search_on_web(config)  # raises SearchRequestError: 429
// after
config.proxy = "http://user:pass@proxy:8080"
results = search_on_web(config)
Defensive patterns

Strategy: try-catch

Validate before calling

import socket
socket.getaddrinfo("www.google.com", 443)  # smoke-test DNS/connectivity first

Try / catch

from scrapegraphai.utils.research_web import SearchRequestError
try:
    results = search_on_web(config)
except SearchRequestError as e:
    logger.warning("search failed: %s", e)
    results = []

Prevention

When it happens

Trigger: Calling search_on_web when the engine endpoint returns an HTTP error (403/429), DNS resolution fails, the connection is reset, or SSL verification fails. Any requests.RequestException other than requests.Timeout triggers it.

Common situations: Scraping Google/Bing HTML endpoints that respond 429/503 to bot traffic; invalid or missing proxy settings; SSL/TLS interception on corporate networks; DNS failures in containers.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

AI-assisted analysis of ScrapeGraphAI/Scrapegraph-ai@532dfffbf6 (2026-08-28). Data as JSON: /api/errors/8df00e5b267c1cc5. Report an issue: GitHub.