{"record":{"id":"9b737f6ff9312c7d","repo":"unclecode/crawl4ai","slug":"unexpected-status-code-for-url","errorCode":null,"errorMessage":"Unexpected status code for {url}","messagePattern":"Unexpected status code for (.+?)","errorType":"http","errorClass":"HTTPStatusError","httpStatus":null,"severity":"error","filePath":"crawl4ai/async_crawler_strategy.py","lineNumber":2724,"sourceCode":"            if config.proxy_config:\n                proxy_url = self._format_proxy_url(config.proxy_config)\n                request_kwargs['proxy'] = proxy_url\n\n            if self.browser_config.method == \"POST\":\n                if self.browser_config.data:\n                    request_kwargs['data'] = self.browser_config.data\n                if self.browser_config.json:\n                    request_kwargs['json'] = self.browser_config.json\n\n            await self.hooks['before_request'](url, request_kwargs)\n\n            try:\n                async with session.request(self.browser_config.method, url, **request_kwargs) as response:\n                    raw_bytes = await response.read()\n                    content = memoryview(raw_bytes)\n\n                    if not (200 <= response.status < 300):\n                        raise HTTPStatusError(\n                            response.status,\n                            f\"Unexpected status code for {url}\"\n                        )\n\n                    response_headers = dict(response.headers)\n                    content_type = response.content_type or 'text/html'\n                    content_type = content_type.split(';')[0].strip().lower()\n                    content_disposition = response_headers.get('Content-Disposition', '')\n\n                    downloaded_files = None\n                    html = \"\"\n\n                    if self._is_file_download(content_type, content_disposition):\n                        # Save file to disk\n                        downloads_path = self.browser_config.downloads_path or os.path.join(\n                            os.path.expanduser(\"~\"), \".crawl4ai\", \"downloads\"\n                        )\n                        os.makedirs(downloads_path, exist_ok=True)","sourceCodeStart":2706,"sourceCodeEnd":2742,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/async_crawler_strategy.py#L2706-L2742","documentation":"Raised by the HTTP-mode crawler when the server responds with a status outside 200-299. It raises HTTPStatusError carrying the numeric status and the URL. The response body has already been read (raw_bytes), but no HTML/AsyncCrawlResponse is produced — non-2xx is treated as failure, with no built-in retry.","triggerScenarios":"Fetching a 404/410 page, 403 from bot protection (Cloudflare), 429 rate limiting, 500/502/503 from origin or proxy, or a redirect chain landing on an error page. Any arun() with AsyncHTTPCrawler/BrowserConfig that selects the HTTP strategy.","commonSituations":"Crawling at high request rates hitting 429s; sites requiring cookies/headers the plain aiohttp request lacks; stale URLs from a sitemap returning 404/410; proxies returning 502/503.","solutions":["Catch HTTPStatusError and branch on e.status: skip 404/410, retry 429/5xx with backoff honoring Retry-After.","Set custom headers (User-Agent, cookies) via BrowserConfig(headers=...) since plain aiohttp is easily blocked.","Reduce request rate / add delays when seeing 429.","If you need the error page's HTML anyway, use the Playwright browser strategy instead of HTTP mode."],"exampleFix":"// before\nresult = await crawler.arun(url)\n\n// after\nfrom crawl4ai.async_crawler_strategy import HTTPStatusError\ntry:\n    result = await crawler.arun(url)\nexcept HTTPStatusError as e:\n    if e.status in (429, 502, 503):\n        await asyncio.sleep(5)\n        result = await crawler.arun(url)\n    else:\n        raise","handlingStrategy":"try-catch","validationCode":"async def url_ok(session, url: str) -> bool:\n    async with session.head(url, allow_redirects=True) as r:\n        return 200 <= r.status < 300","typeGuard":null,"tryCatchPattern":"from crawl4ai.async_crawler_strategy import HTTPStatusError\n\ntry:\n    result = await crawler.arun(url)\nexcept HTTPStatusError as e:\n    if e.status in (404, 410):\n        mark_gone(url)\n    elif e.status in (429, 502, 503):\n        await asyncio.sleep(5)\n        result = await crawler.arun(url)\n    else:\n        raise","preventionTips":["Send realistic User-Agent and headers in HTTP mode","Honor Retry-After on 429","Prune dead URLs from sitemaps periodically"],"tags":["http-status","http-crawler","network"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}