{"record":{"id":"8df00e5b267c1cc5","repo":"ScrapeGraphAI/Scrapegraph-ai","slug":"search-request-failed-str-e","errorCode":null,"errorMessage":"Search request failed: {str(e)}","messagePattern":"Search request failed: (.+?)","errorType":"exception","errorClass":"SearchRequestError","httpStatus":null,"severity":"error","filePath":"scrapegraphai/utils/research_web.py","lineNumber":241,"sourceCode":"                config.query, config.max_results, config.timeout, formatted_proxy\n            )\n\n        elif config.search_engine == \"searxng\":\n            results = _search_searxng(\n                config.query, config.max_results, config.port, config.timeout\n            )\n\n        elif config.search_engine == \"serper\":\n            results = _search_serper(\n                config.query, config.max_results, config.serper_api_key, config.timeout\n            )\n\n        return filter_pdf_links(results)\n\n    except requests.Timeout:\n        raise TimeoutError(f\"Search request timed out after {timeout} seconds\")\n    except requests.RequestException as e:\n        raise SearchRequestError(f\"Search request failed: {str(e)}\")\n    except ValueError as e:\n        raise SearchConfigError(f\"Invalid search configuration: {str(e)}\")\n\n\ndef _search_duckduckgo(\n    query: str, max_results: int, proxy: Optional[str] = None\n) -> List[str]:\n    \"\"\"\n    Helper function for DuckDuckGo search using the ``ddgs`` package.\n\n    The ``duckduckgo-search`` package was renamed to ``ddgs``; recent\n    ``langchain-community`` releases import ``from ddgs import DDGS``, which\n    silently broke the previous langchain-based implementation. This calls\n    ``ddgs`` directly so results no longer depend on parsing a formatted string.\n\n    Args:\n        query (str): Search query\n        max_results (int): Maximum number of results to return","sourceCodeStart":223,"sourceCodeEnd":259,"githubUrl":"https://github.com/ScrapeGraphAI/Scrapegraph-ai/blob/532dfffbf6ee823a6c9cf8cfedc24a93bf026780/scrapegraphai/utils/research_web.py#L223-L259","documentation":"Raised by search_on_web when a search request fails with a non-timeout requests exception (connection error, HTTP error, malformed URL, TLS failure). It wraps the original requests.RequestException message into SearchRequestError for a uniform error surface.","triggerScenarios":"Calling search_on_web when the engine endpoint returns an HTTP error (403/429), DNS resolution fails, the connection is reset, or SSL verification fails. Any requests.RequestException other than requests.Timeout triggers it.","commonSituations":"Scraping Google/Bing HTML endpoints that respond 429/503 to bot traffic; invalid or missing proxy settings; SSL/TLS interception on corporate networks; DNS failures in containers.","solutions":["Inspect the wrapped message (str(e)) to identify the HTTP/connection cause","Add or fix proxy configuration in the SearchConfig","If 429/403, reduce request frequency or switch to an API-based engine (serper)","For SSL issues, verify certificates or configure the environment's CA bundle"],"exampleFix":"// before\nresults = search_on_web(config)  # raises SearchRequestError: 429\n// after\nconfig.proxy = \"http://user:pass@proxy:8080\"\nresults = search_on_web(config)","handlingStrategy":"try-catch","validationCode":"import socket\nsocket.getaddrinfo(\"www.google.com\", 443)  # smoke-test DNS/connectivity first","typeGuard":null,"tryCatchPattern":"from scrapegraphai.utils.research_web import SearchRequestError\ntry:\n    results = search_on_web(config)\nexcept SearchRequestError as e:\n    logger.warning(\"search failed: %s\", e)\n    results = []","preventionTips":["Configure a proxy for restricted networks","Catch SearchRequestError and fall back to another engine","Rate-limit repeated scraping calls to avoid 429/403"],"tags":["network","http","web-search"],"backgroundTag":"http-request-failed","analyzedSha":"532dfffbf6ee823a6c9cf8cfedc24a93bf026780","analyzedAt":"2026-08-28T15:19:38.821Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}