unclecode/crawl4ai · error · HTTPCrawlerError

HTTP request failed: {str(e)}

Error message

HTTP request failed: {str(e)}

What it means

The catch-all handler in the HTTP-mode crawler: any exception that is not ServerTimeoutError, ClientConnectorError, ClientError, or asyncio.TimeoutError is wrapped in HTTPCrawlerError with 'HTTP request failed'. This captures unexpected failures such as decoding errors, SSL context problems surfaced as non-aiohttp exceptions, or bugs in user hooks registered on 'before_request'/'after_request'.

Source

Thrown at crawl4ai/async_crawler_strategy.py:2796

            except aiohttp.ServerTimeoutError as e:
                await self.hooks['on_error'](e)
                raise ConnectionTimeoutError(f"Request timed out: {str(e)}")
                
            except aiohttp.ClientConnectorError as e:
                await self.hooks['on_error'](e)
                raise ConnectionError(f"Connection failed: {str(e)}")
                
            except aiohttp.ClientError as e:
                await self.hooks['on_error'](e)
                raise HTTPCrawlerError(f"HTTP client error: {str(e)}")
            
            except asyncio.exceptions.TimeoutError as e:
                await self.hooks['on_error'](e)
                raise ConnectionTimeoutError(f"Request timed out: {str(e)}")
            
            except Exception as e:
                await self.hooks['on_error'](e)
                raise HTTPCrawlerError(f"HTTP request failed: {str(e)}")

    async def crawl(
        self, 
        url: str, 
        config: Optional[CrawlerRunConfig] = None, 
        **kwargs
    ) -> AsyncCrawlResponse:
        config = config or CrawlerRunConfig.from_kwargs(kwargs)
        
        parsed = urlparse(url)
        scheme = parsed.scheme.rstrip('/')
        
        if scheme not in self.VALID_SCHEMES:
            raise ValueError(f"Unsupported URL scheme: {scheme}")
            
        try:
            if scheme == 'file':
                return await self._handle_file(parsed.path)

View on GitHub (pinned to 7e80152142)

Solutions

  1. Read the wrapped original exception text — it names the real failing operation.
  2. Audit custom hooks registered via set_hook('before_request'/'after_request') for exceptions and argument mutation errors.
  3. Reproduce the URL with plain aiohttp in a script to isolate whether crawl4ai or the environment is at fault.
  4. Wrap the crawl call in try/except HTTPCrawlerError to keep batch runs alive.

Example fix

// before
strategy.set_hook('before_request', lambda url, kw: kw.update(timeout='30'))  # wrong type, raises

// after
async def before_request(url, request_kwargs):
    request_kwargs['timeout'] = 30
strategy.set_hook('before_request', before_request)
Defensive patterns

Strategy: try-catch

Try / catch

from crawl4ai.async_crawler_strategy import HTTPCrawlerError

try:
    result = await crawler.arun(url)
except HTTPCrawlerError as e:
    logger.exception(f"unexpected crawl failure for {url}")
    raise

Prevention

When it happens

Trigger: A user hook raising an arbitrary exception during before_request/after_request (on_error hook runs first, then re-raised wrapped); response body decoding issues; environment-level SSL module errors; any programming error inside the request path.

Common situations: Custom before_request hooks that mutate request_kwargs incorrectly (e.g. setting a non-serializable value) and raise; exotic TLS environments; bugs that only surface for specific URLs.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/6764a233dccca20c. Report an issue: GitHub.