unclecode/crawl4ai · error · HTTPCrawlerError

HTTP client error: {str(e)}

Error message

HTTP client error: {str(e)}

What it means

Raised when aiohttp raises a generic ClientError (not the more specific timeout/connector subclasses) during an HTTP-mode crawl. This covers mid-request protocol failures: chunked encoding errors, invalid HTTP responses, connection resets after establishment, redirect loops, and cookie/ header processing errors. It is wrapped in HTTPCrawlerError with 'HTTP client error'.

Source

Thrown at crawl4ai/async_crawler_strategy.py:2788

                        status_code=response.status,
                        redirected_url=str(response.url),
                        downloaded_files=downloaded_files,
                    )

                    await self.hooks['after_request'](result)
                    return result

            except aiohttp.ServerTimeoutError as e:
                await self.hooks['on_error'](e)
                raise ConnectionTimeoutError(f"Request timed out: {str(e)}")
                
            except aiohttp.ClientConnectorError as e:
                await self.hooks['on_error'](e)
                raise ConnectionError(f"Connection failed: {str(e)}")
                
            except aiohttp.ClientError as e:
                await self.hooks['on_error'](e)
                raise HTTPCrawlerError(f"HTTP client error: {str(e)}")
            
            except asyncio.exceptions.TimeoutError as e:
                await self.hooks['on_error'](e)
                raise ConnectionTimeoutError(f"Request timed out: {str(e)}")
            
            except Exception as e:
                await self.hooks['on_error'](e)
                raise HTTPCrawlerError(f"HTTP request failed: {str(e)}")

    async def crawl(
        self, 
        url: str, 
        config: Optional[CrawlerRunConfig] = None, 
        **kwargs
    ) -> AsyncCrawlResponse:
        config = config or CrawlerRunConfig.from_kwargs(kwargs)
        
        parsed = urlparse(url)

View on GitHub (pinned to 7e80152142)

Solutions

  1. Inspect the embedded aiohttp message to identify the protocol-level cause.
  2. Retry once with a fresh session — mid-request resets are often transient, especially with connection reuse.
  3. Try the Playwright browser strategy instead: a real browser tolerates quirky HTTP that aiohttp rejects.
  4. Disable keep-alive or cap redirects if the message points at those mechanisms.

Example fix

// before
result = await crawler.arun(url)

// after
from crawl4ai.async_crawler_strategy import HTTPCrawlerError
try:
    result = await crawler.arun(url)
except HTTPCrawlerError as e:
    logger.warning(f"http client error on {url}: {e}; retrying via browser")
    async with AsyncWebCrawler() as bc:  # default Playwright strategy
        result = await bc.arun(url)
Defensive patterns

Strategy: fallback

Try / catch

from crawl4ai.async_crawler_strategy import HTTPCrawlerError

try:
    result = await crawler.arun(url)
except HTTPCrawlerError as e:
    if "HTTP client error" in str(e):
        async with AsyncWebCrawler() as browser_crawler:
            result = await browser_crawler.arun(url)

Prevention

When it happens

Trigger: Server sends malformed HTTP or truncated chunked responses; connection reset mid-body; too many redirects (aiohttp raises ClientHttpProxyError or similar under this class); responses with invalid headers the client rejects.

Common situations: Crawling misbehaving or bot-protected origin servers behind CDNs; old servers with non-conforming HTTP; aggressive keep-alive reuse against servers that drop idle connections; redirect chains exceeding limits.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/17559603cf8afb17. Report an issue: GitHub.