{"record":{"id":"9cecb421358651da","repo":"ScrapeGraphAI/Scrapegraph-ai","slug":"bing-search-failed-str-e","errorCode":null,"errorMessage":"Bing search failed: {str(e)}","messagePattern":"Bing search failed: (.+?)","errorType":"exception","errorClass":"SearchRequestError","httpStatus":null,"severity":"error","filePath":"scrapegraphai/utils/research_web.py","lineNumber":329,"sourceCode":"            proxies=proxies,\n            timeout=timeout,\n        )\n        response.raise_for_status()\n\n        soup = BeautifulSoup(response.text, \"html.parser\")\n        results = []\n\n        # Extract URLs from Bing search results\n        for link in soup.select(\"li.b_algo h2 a\"):\n            url = link.get(\"href\")\n            if url and url.startswith(\"http\"):\n                results.append(url)\n                if len(results) >= max_results:\n                    break\n\n        return results\n    except Exception as e:\n        raise SearchRequestError(f\"Bing search failed: {str(e)}\")\n\n\ndef _search_searxng(query: str, max_results: int, port: int, timeout: int) -> List[str]:\n    \"\"\"\n    Helper function for SearXNG search.\n\n    Args:\n        query (str): Search query\n        max_results (int): Maximum number of results to return\n        port (int): Port for SearXNG\n        timeout (int): Request timeout in seconds\n\n    Returns:\n        List[str]: List of URLs from search results\n    \"\"\"\n    headers = {\"User-Agent\": get_random_user_agent()}\n\n    params = {","sourceCodeStart":311,"sourceCodeEnd":347,"githubUrl":"https://github.com/ScrapeGraphAI/Scrapegraph-ai/blob/532dfffbf6ee823a6c9cf8cfedc24a93bf026780/scrapegraphai/utils/research_web.py#L311-L347","documentation":"Raised by the Bing backend when scraping Bing's HTML search results fails with any exception (HTTP error, parse failure, missing elements). The original exception message is wrapped into SearchRequestError.","triggerScenarios":"Calling search_on_web with search_engine='bing' when Bing returns an error page, changes its HTML markup, blocks the request, or the response cannot be parsed.","commonSituations":"Bing markup changes breaking the scraper; bot detection returning a challenge page; regional Bing redirects altering the DOM.","solutions":["Check str(e) for the underlying cause (HTTP status vs parse error)","Switch to an API-based engine (serper) for reliability","Update scrapegraphai — scraper fixes often land in patches","Add a proxy or adjust headers via proxy config if blocked"],"exampleFix":"// before\nconfig.search_engine = \"bing\"  # scraping fails\n// after\nconfig.search_engine = \"serper\"\nconfig.serper_api_key = os.getenv(\"SERPER_API_KEY\")","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    results = search_on_web(config)\nexcept SearchRequestError as e:\n    if \"Bing\" in str(e):\n        config.search_engine = \"serper\"; config.serper_api_key = os.getenv(\"SERPER_APIKEY\")\n        results = search_on_web(config)\n    else:\n        raise","preventionTips":["Prefer API-based engines over HTML scraping in production","Keep scrapegraphai updated for scraper fixes"],"tags":["web-search","scraping","bing"],"backgroundTag":"web-scraping-blocked","analyzedSha":"532dfffbf6ee823a6c9cf8cfedc24a93bf026780","analyzedAt":"2026-08-28T15:19:38.821Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}