unclecode/crawl4ai · error · HTTPStatusError

Unexpected status code for {url}

Error message

Unexpected status code for {url}

What it means

Raised by the HTTP-mode crawler when the server responds with a status outside 200-299. It raises HTTPStatusError carrying the numeric status and the URL. The response body has already been read (raw_bytes), but no HTML/AsyncCrawlResponse is produced — non-2xx is treated as failure, with no built-in retry.

Source

Thrown at crawl4ai/async_crawler_strategy.py:2724

            if config.proxy_config:
                proxy_url = self._format_proxy_url(config.proxy_config)
                request_kwargs['proxy'] = proxy_url

            if self.browser_config.method == "POST":
                if self.browser_config.data:
                    request_kwargs['data'] = self.browser_config.data
                if self.browser_config.json:
                    request_kwargs['json'] = self.browser_config.json

            await self.hooks['before_request'](url, request_kwargs)

            try:
                async with session.request(self.browser_config.method, url, **request_kwargs) as response:
                    raw_bytes = await response.read()
                    content = memoryview(raw_bytes)

                    if not (200 <= response.status < 300):
                        raise HTTPStatusError(
                            response.status,
                            f"Unexpected status code for {url}"
                        )

                    response_headers = dict(response.headers)
                    content_type = response.content_type or 'text/html'
                    content_type = content_type.split(';')[0].strip().lower()
                    content_disposition = response_headers.get('Content-Disposition', '')

                    downloaded_files = None
                    html = ""

                    if self._is_file_download(content_type, content_disposition):
                        # Save file to disk
                        downloads_path = self.browser_config.downloads_path or os.path.join(
                            os.path.expanduser("~"), ".crawl4ai", "downloads"
                        )
                        os.makedirs(downloads_path, exist_ok=True)

View on GitHub (pinned to 7e80152142)

Solutions

  1. Catch HTTPStatusError and branch on e.status: skip 404/410, retry 429/5xx with backoff honoring Retry-After.
  2. Set custom headers (User-Agent, cookies) via BrowserConfig(headers=...) since plain aiohttp is easily blocked.
  3. Reduce request rate / add delays when seeing 429.
  4. If you need the error page's HTML anyway, use the Playwright browser strategy instead of HTTP mode.

Example fix

// before
result = await crawler.arun(url)

// after
from crawl4ai.async_crawler_strategy import HTTPStatusError
try:
    result = await crawler.arun(url)
except HTTPStatusError as e:
    if e.status in (429, 502, 503):
        await asyncio.sleep(5)
        result = await crawler.arun(url)
    else:
        raise
Defensive patterns

Strategy: try-catch

Validate before calling

async def url_ok(session, url: str) -> bool:
    async with session.head(url, allow_redirects=True) as r:
        return 200 <= r.status < 300

Try / catch

from crawl4ai.async_crawler_strategy import HTTPStatusError

try:
    result = await crawler.arun(url)
except HTTPStatusError as e:
    if e.status in (404, 410):
        mark_gone(url)
    elif e.status in (429, 502, 503):
        await asyncio.sleep(5)
        result = await crawler.arun(url)
    else:
        raise

Prevention

When it happens

Trigger: Fetching a 404/410 page, 403 from bot protection (Cloudflare), 429 rate limiting, 500/502/503 from origin or proxy, or a redirect chain landing on an error page. Any arun() with AsyncHTTPCrawler/BrowserConfig that selects the HTTP strategy.

Common situations: Crawling at high request rates hitting 429s; sites requiring cookies/headers the plain aiohttp request lacks; stale URLs from a sitemap returning 404/410; proxies returning 502/503.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/9b737f6ff9312c7d. Report an issue: GitHub.