ScrapeGraphAI/Scrapegraph-ai · error · SearchRequestError

DuckDuckGo search failed: {str(e)}

Error message

DuckDuckGo search failed: {str(e)}

What it means

Raised by _search_duckduckgo when the ddgs text search raises any exception (rate limit, network failure, empty/malformed response). It is wrapped into SearchRequestError with the original message preserved.

Source

Thrown at scrapegraphai/utils/research_web.py:282

    """
    try:
        from ddgs import DDGS
    except ImportError as e:
        raise ImportError(
            "Could not import the 'ddgs' package required for DuckDuckGo "
            "search. Please install it with `pip install -U ddgs`."
        ) from e

    try:
        with DDGS(proxy=proxy) as ddgs:
            results = [
                result["href"]
                for result in ddgs.text(query, max_results=max_results)
                if result.get("href")
            ]
        return results
    except Exception as e:
        raise SearchRequestError(f"DuckDuckGo search failed: {str(e)}")


def _search_bing(
    query: str, max_results: int, timeout: int, proxy: Optional[str] = None
) -> List[str]:
    """
    Helper function for Bing search with improved error handling.

    Args:
        query (str): Search query
        max_results (int): Maximum number of results to return
        timeout (int): Request timeout in seconds
        proxy (str, optional): Proxy configuration

    Returns:
        List[str]: List of URLs from search results
    """
    headers = {"User-Agent": get_random_user_agent()}

View on GitHub (pinned to 532dfffbf6)

Solutions

1. Slow down / add delays between DuckDuckGo searches
2. Configure a proxy in the SearchConfig
3. Update the ddgs package to the latest version
4. Fall back to another engine (serper, searxng) when this error occurs

Example fix

// before
results = search_on_web(config)  # engine='duckduckgo', rate-limited
// after
try:
    results = search_on_web(config)
except SearchRequestError:
    config.search_engine = "serper"
    results = search_on_web(config)

try:
    results = search_on_web(config)
except SearchRequestError:
    config.search_engine = "google"
    results = search_on_web(config)

When it happens

Trigger: Calling search_on_web with engine='duckduckgo' when DuckDuckGo rate-limits the client, the proxy blocks duckduckgo.com, or ddgs returns an unexpected response shape.

Common situations: Running many search queries from one IP triggering DuckDuckGo rate limits; datacenter IPs being blocked; ddgs package version changes altering result keys.

Related errors

• missing proxy servers for criteria
• Search request timed out after {timeout} seconds
• Search request failed: {str(e)}
• Could not import the 'ddgs' package required for DuckDuckGo
• SearXNG search failed: {str(e)}

AI-assisted analysis of ScrapeGraphAI/Scrapegraph-ai@532dfffbf6 (2026-08-28). Data as JSON: /api/errors/e6759036b0e9b43d. Report an issue: GitHub.