unclecode/crawl4ai · error · ConnectionError

Connection failed: {str(e)}

Error message

Connection failed: {str(e)}

What it means

Raised when aiohttp raises ClientConnectorError, meaning the TCP/TLS connection could never be established: DNS resolution failure, refused connection, unreachable host, or TLS handshake failure. The HTTP crawler wraps it into a plain ConnectionError with the underlying cause text.

Source

Thrown at crawl4ai/async_crawler_strategy.py:2784

                    result = AsyncCrawlResponse(
                        html=html,
                        response_headers=response_headers,
                        status_code=response.status,
                        redirected_url=str(response.url),
                        downloaded_files=downloaded_files,
                    )

                    await self.hooks['after_request'](result)
                    return result

            except aiohttp.ServerTimeoutError as e:
                await self.hooks['on_error'](e)
                raise ConnectionTimeoutError(f"Request timed out: {str(e)}")
                
            except aiohttp.ClientConnectorError as e:
                await self.hooks['on_error'](e)
                raise ConnectionError(f"Connection failed: {str(e)}")
                
            except aiohttp.ClientError as e:
                await self.hooks['on_error'](e)
                raise HTTPCrawlerError(f"HTTP client error: {str(e)}")
            
            except asyncio.exceptions.TimeoutError as e:
                await self.hooks['on_error'](e)
                raise ConnectionTimeoutError(f"Request timed out: {str(e)}")
            
            except Exception as e:
                await self.hooks['on_error'](e)
                raise HTTPCrawlerError(f"HTTP request failed: {str(e)}")

    async def crawl(
        self, 
        url: str, 
        config: Optional[CrawlerRunConfig] = None, 
        **kwargs

View on GitHub (pinned to 7e80152142)

Solutions

  1. Fix or verify the URL/host and check DNS (dig/nslookup) from the same environment.
  2. For local targets, confirm the dev server is up and the port matches.
  3. Configure proxy settings if the environment requires an egress proxy.
  4. Catch ConnectionError per URL during batch crawls and mark the URL dead rather than aborting the run.

Example fix

// before
results = [await crawler.arun(u) for u in urls]  # one dead host kills the batch

// after
results = []
for u in urls:
    try:
        results.append(await crawler.arun(u))
    except ConnectionError as e:
        logger.warning(f"unreachable, skipping {u}: {e}")
Defensive patterns

Strategy: try-catch

Validate before calling

import socket
from urllib.parse import urlparse

def host_resolves(url: str) -> bool:
    host = urlparse(url).hostname
    if not host:
        return False
    try:
        socket.gethostbyname(host)
        return True
    except socket.gaierror:
        return False

Try / catch

try:
    result = await crawler.arun(url)
except ConnectionError as e:
    if "Connection failed" in str(e):
        mark_unreachable(url)

Prevention

When it happens

Trigger: Crawling a URL whose host does not resolve (typo, dead domain); target port not listening (dev server down); firewall/proxy blocking egress; TLS certificate rejected at handshake; IPv6-only host without IPv6 connectivity.

Common situations: Crawling localhost during local development when the dev server is not running; CI environments without network access or with DNS restrictions; expired domains in stale URL lists; corporate proxies requiring explicit configuration.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/539c080c26970888. Report an issue: GitHub.