{"record":{"id":"17559603cf8afb17","repo":"unclecode/crawl4ai","slug":"http-client-error-str-e","errorCode":null,"errorMessage":"HTTP client error: {str(e)}","messagePattern":"HTTP client error: (.+?)","errorType":"exception","errorClass":"HTTPCrawlerError","httpStatus":null,"severity":"error","filePath":"crawl4ai/async_crawler_strategy.py","lineNumber":2788,"sourceCode":"                        status_code=response.status,\n                        redirected_url=str(response.url),\n                        downloaded_files=downloaded_files,\n                    )\n\n                    await self.hooks['after_request'](result)\n                    return result\n\n            except aiohttp.ServerTimeoutError as e:\n                await self.hooks['on_error'](e)\n                raise ConnectionTimeoutError(f\"Request timed out: {str(e)}\")\n                \n            except aiohttp.ClientConnectorError as e:\n                await self.hooks['on_error'](e)\n                raise ConnectionError(f\"Connection failed: {str(e)}\")\n                \n            except aiohttp.ClientError as e:\n                await self.hooks['on_error'](e)\n                raise HTTPCrawlerError(f\"HTTP client error: {str(e)}\")\n            \n            except asyncio.exceptions.TimeoutError as e:\n                await self.hooks['on_error'](e)\n                raise ConnectionTimeoutError(f\"Request timed out: {str(e)}\")\n            \n            except Exception as e:\n                await self.hooks['on_error'](e)\n                raise HTTPCrawlerError(f\"HTTP request failed: {str(e)}\")\n\n    async def crawl(\n        self, \n        url: str, \n        config: Optional[CrawlerRunConfig] = None, \n        **kwargs\n    ) -> AsyncCrawlResponse:\n        config = config or CrawlerRunConfig.from_kwargs(kwargs)\n        \n        parsed = urlparse(url)","sourceCodeStart":2770,"sourceCodeEnd":2806,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/async_crawler_strategy.py#L2770-L2806","documentation":"Raised when aiohttp raises a generic ClientError (not the more specific timeout/connector subclasses) during an HTTP-mode crawl. This covers mid-request protocol failures: chunked encoding errors, invalid HTTP responses, connection resets after establishment, redirect loops, and cookie/ header processing errors. It is wrapped in HTTPCrawlerError with 'HTTP client error'.","triggerScenarios":"Server sends malformed HTTP or truncated chunked responses; connection reset mid-body; too many redirects (aiohttp raises ClientHttpProxyError or similar under this class); responses with invalid headers the client rejects.","commonSituations":"Crawling misbehaving or bot-protected origin servers behind CDNs; old servers with non-conforming HTTP; aggressive keep-alive reuse against servers that drop idle connections; redirect chains exceeding limits.","solutions":["Inspect the embedded aiohttp message to identify the protocol-level cause.","Retry once with a fresh session — mid-request resets are often transient, especially with connection reuse.","Try the Playwright browser strategy instead: a real browser tolerates quirky HTTP that aiohttp rejects.","Disable keep-alive or cap redirects if the message points at those mechanisms."],"exampleFix":"// before\nresult = await crawler.arun(url)\n\n// after\nfrom crawl4ai.async_crawler_strategy import HTTPCrawlerError\ntry:\n    result = await crawler.arun(url)\nexcept HTTPCrawlerError as e:\n    logger.warning(f\"http client error on {url}: {e}; retrying via browser\")\n    async with AsyncWebCrawler() as bc:  # default Playwright strategy\n        result = await bc.arun(url)","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"from crawl4ai.async_crawler_strategy import HTTPCrawlerError\n\ntry:\n    result = await crawler.arun(url)\nexcept HTTPCrawlerError as e:\n    if \"HTTP client error\" in str(e):\n        async with AsyncWebCrawler() as browser_crawler:\n            result = await browser_crawler.arun(url)","preventionTips":["Keep a browser-strategy fallback for quirky servers","Retry once with a fresh session for reset errors","Log the embedded aiohttp cause for diagnosis"],"tags":["http-client","aiohttp","network","http-crawler"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}