ScrapeGraphAI/Scrapegraph-ai · error · SearchRequestError

Bing search failed: {str(e)}

Error message

Bing search failed: {str(e)}

What it means

Raised by the Bing backend when scraping Bing's HTML search results fails with any exception (HTTP error, parse failure, missing elements). The original exception message is wrapped into SearchRequestError.

Source

Thrown at scrapegraphai/utils/research_web.py:329

            proxies=proxies,
            timeout=timeout,
        )
        response.raise_for_status()

        soup = BeautifulSoup(response.text, "html.parser")
        results = []

        # Extract URLs from Bing search results
        for link in soup.select("li.b_algo h2 a"):
            url = link.get("href")
            if url and url.startswith("http"):
                results.append(url)
                if len(results) >= max_results:
                    break

        return results
    except Exception as e:
        raise SearchRequestError(f"Bing search failed: {str(e)}")


def _search_searxng(query: str, max_results: int, port: int, timeout: int) -> List[str]:
    """
    Helper function for SearXNG search.

    Args:
        query (str): Search query
        max_results (int): Maximum number of results to return
        port (int): Port for SearXNG
        timeout (int): Request timeout in seconds

    Returns:
        List[str]: List of URLs from search results
    """
    headers = {"User-Agent": get_random_user_agent()}

    params = {

View on GitHub (pinned to 532dfffbf6)

Solutions

1. Check str(e) for the underlying cause (HTTP status vs parse error)
2. Switch to an API-based engine (serper) for reliability
3. Update scrapegraphai — scraper fixes often land in patches
4. Add a proxy or adjust headers via proxy config if blocked

Example fix

// before
config.search_engine = "bing"  # scraping fails
// after
config.search_engine = "serper"
config.serper_api_key = os.getenv("SERPER_API_KEY")

try:
    results = search_on_web(config)
except SearchRequestError as e:
    if "Bing" in str(e):
        config.search_engine = "serper"; config.serper_api_key = os.getenv("SERPER_APIKEY")
        results = search_on_web(config)
    else:
        raise

When it happens

Trigger: Calling search_on_web with search_engine='bing' when Bing returns an error page, changes its HTML markup, blocks the request, or the response cannot be parsed.

Common situations: Bing markup changes breaking the scraper; bot detection returning a challenge page; regional Bing redirects altering the DOM.

Related errors

• Unsupported backend: {self.backend}
• undetected_chromedriver is required for ChromiumLoader. Plea
• Search request timed out after {timeout} seconds
• Search request failed: {str(e)}
• Invalid search configuration: {str(e)}

AI-assisted analysis of ScrapeGraphAI/Scrapegraph-ai@532dfffbf6 (2026-08-28). Data as JSON: /api/errors/9cecb421358651da. Report an issue: GitHub.