ScrapeGraphAI/Scrapegraph-ai · error · TimeoutError

Search request timed out after {timeout} seconds

Error message

Search request timed out after {timeout} seconds

What it means

Raised by search_on_web when the underlying HTTP search request exceeds the configured timeout and requests.Timeout is caught. It is re-raised as a built-in TimeoutError so callers can handle search latency uniformly across engines. The timeout value comes from the SearchConfig passed to search_on_web.

Source

Thrown at scrapegraphai/utils/research_web.py:239

        elif config.search_engine == "bing":
            results = _search_bing(
                config.query, config.max_results, config.timeout, formatted_proxy
            )

        elif config.search_engine == "searxng":
            results = _search_searxng(
                config.query, config.max_results, config.port, config.timeout
            )

        elif config.search_engine == "serper":
            results = _search_serper(
                config.query, config.max_results, config.serper_api_key, config.timeout
            )

        return filter_pdf_links(results)

    except requests.Timeout:
        raise TimeoutError(f"Search request timed out after {timeout} seconds")
    except requests.RequestException as e:
        raise SearchRequestError(f"Search request failed: {str(e)}")
    except ValueError as e:
        raise SearchConfigError(f"Invalid search configuration: {str(e)}")


def _search_duckduckgo(
    query: str, max_results: int, proxy: Optional[str] = None
) -> List[str]:
    """
    Helper function for DuckDuckGo search using the ``ddgs`` package.

    The ``duckduckgo-search`` package was renamed to ``ddgs``; recent
    ``langchain-community`` releases import ``from ddgs import DDGS``, which
    silently broke the previous langchain-based implementation. This calls
    ``ddgs`` directly so results no longer depend on parsing a formatted string.

    Args:

View on GitHub (pinned to 532dfffbf6)

Solutions

1. Increase config.timeout (e.g. 15-30 seconds) and retry
2. Check network/proxy connectivity to the search endpoint
3. Wrap search_on_web in a retry with exponential backoff for transient slowness
4. If consistently timing out, switch to a faster engine (e.g. serper with a valid API key)

Example fix

// before
results = search_on_web(SearchConfig(query="x", timeout=3))
// after
results = search_on_web(SearchConfig(query="x", timeout=30))

if config.timeout < 10:
    config.timeout = 15  # raise floor before calling

try:
    results = search_on_web(config)
except TimeoutError:
    config.timeout *= 2
    results = search_on_web(config)

When it happens

Trigger: Calling search_on_web (directly or via the research/execute graph step) with a short config.timeout while the search engine endpoint (Google/Bing/SearXNG/Serper) is slow, unreachable, or blocked by a proxy/firewall.

Common situations: Corporate proxies or restricted networks slowing requests; setting timeout too low (default few seconds); rate-limited or throttled search endpoints; running in CI without network access.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Search request failed: {str(e)}
• undetected_chromedriver is required for ChromiumLoader. Plea
• PDF parsing exceeded timeout of {self.timeout} seconds
• Response took longer than {timeout} seconds
• missing proxy servers for criteria

AI-assisted analysis of ScrapeGraphAI/Scrapegraph-ai@532dfffbf6 (2026-08-28). Data as JSON: /api/errors/a2304133e2ad76ce. Report an issue: GitHub.