{"record":{"id":"6764a233dccca20c","repo":"unclecode/crawl4ai","slug":"http-request-failed-str-e","errorCode":null,"errorMessage":"HTTP request failed: {str(e)}","messagePattern":"HTTP request failed: (.+?)","errorType":"exception","errorClass":"HTTPCrawlerError","httpStatus":null,"severity":"error","filePath":"crawl4ai/async_crawler_strategy.py","lineNumber":2796,"sourceCode":"            except aiohttp.ServerTimeoutError as e:\n                await self.hooks['on_error'](e)\n                raise ConnectionTimeoutError(f\"Request timed out: {str(e)}\")\n                \n            except aiohttp.ClientConnectorError as e:\n                await self.hooks['on_error'](e)\n                raise ConnectionError(f\"Connection failed: {str(e)}\")\n                \n            except aiohttp.ClientError as e:\n                await self.hooks['on_error'](e)\n                raise HTTPCrawlerError(f\"HTTP client error: {str(e)}\")\n            \n            except asyncio.exceptions.TimeoutError as e:\n                await self.hooks['on_error'](e)\n                raise ConnectionTimeoutError(f\"Request timed out: {str(e)}\")\n            \n            except Exception as e:\n                await self.hooks['on_error'](e)\n                raise HTTPCrawlerError(f\"HTTP request failed: {str(e)}\")\n\n    async def crawl(\n        self, \n        url: str, \n        config: Optional[CrawlerRunConfig] = None, \n        **kwargs\n    ) -> AsyncCrawlResponse:\n        config = config or CrawlerRunConfig.from_kwargs(kwargs)\n        \n        parsed = urlparse(url)\n        scheme = parsed.scheme.rstrip('/')\n        \n        if scheme not in self.VALID_SCHEMES:\n            raise ValueError(f\"Unsupported URL scheme: {scheme}\")\n            \n        try:\n            if scheme == 'file':\n                return await self._handle_file(parsed.path)","sourceCodeStart":2778,"sourceCodeEnd":2814,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/async_crawler_strategy.py#L2778-L2814","documentation":"The catch-all handler in the HTTP-mode crawler: any exception that is not ServerTimeoutError, ClientConnectorError, ClientError, or asyncio.TimeoutError is wrapped in HTTPCrawlerError with 'HTTP request failed'. This captures unexpected failures such as decoding errors, SSL context problems surfaced as non-aiohttp exceptions, or bugs in user hooks registered on 'before_request'/'after_request'.","triggerScenarios":"A user hook raising an arbitrary exception during before_request/after_request (on_error hook runs first, then re-raised wrapped); response body decoding issues; environment-level SSL module errors; any programming error inside the request path.","commonSituations":"Custom before_request hooks that mutate request_kwargs incorrectly (e.g. setting a non-serializable value) and raise; exotic TLS environments; bugs that only surface for specific URLs.","solutions":["Read the wrapped original exception text — it names the real failing operation.","Audit custom hooks registered via set_hook('before_request'/'after_request') for exceptions and argument mutation errors.","Reproduce the URL with plain aiohttp in a script to isolate whether crawl4ai or the environment is at fault.","Wrap the crawl call in try/except HTTPCrawlerError to keep batch runs alive."],"exampleFix":"// before\nstrategy.set_hook('before_request', lambda url, kw: kw.update(timeout='30'))  # wrong type, raises\n\n// after\nasync def before_request(url, request_kwargs):\n    request_kwargs['timeout'] = 30\nstrategy.set_hook('before_request', before_request)","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"from crawl4ai.async_crawler_strategy import HTTPCrawlerError\n\ntry:\n    result = await crawler.arun(url)\nexcept HTTPCrawlerError as e:\n    logger.exception(f\"unexpected crawl failure for {url}\")\n    raise","preventionTips":["Keep before_request/after_request hooks exception-free","Log the wrapped cause, not just the wrapper","Reproduce failing URLs with plain aiohttp to isolate the fault"],"tags":["http-crawler","hooks","unexpected-error","network"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}