{"record":{"id":"538c80c73e8ce53b","repo":"unclecode/crawl4ai","slug":"http-result-status-code-error-for-url-urls-0","errorCode":null,"errorMessage":"HTTP {result.status_code} error for URL '{urls[0]}'","messagePattern":"HTTP (.+?) error for URL '(.+?)'","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"crawl4ai/extraction_strategy.py","lineNumber":1847,"sourceCode":"            from .async_configs import BrowserConfig, CrawlerRunConfig, CacheMode\n\n            browser_config = BrowserConfig(\n                headless=True,\n                text_mode=True,\n                light_mode=True,\n            )\n            crawler_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)\n\n            # Normalize to list\n            urls = [url] if isinstance(url, str) else url\n\n            async with AsyncWebCrawler(config=browser_config) as crawler:\n                if len(urls) == 1:\n                    result = await crawler.arun(url=urls[0], config=crawler_config)\n                    if not result.success:\n                        raise Exception(f\"Failed to fetch URL '{urls[0]}': {result.error_message}\")\n                    if result.status_code >= 400:\n                        raise Exception(f\"HTTP {result.status_code} error for URL '{urls[0]}'\")\n                    html = result.html\n                    original_htmls = [result.html]\n                else:\n                    results = await crawler.arun_many(urls=urls, config=crawler_config)\n                    html_parts = []\n                    for i, result in enumerate(results, 1):\n                        if not result.success:\n                            raise Exception(f\"Failed to fetch URL '{result.url}': {result.error_message}\")\n                        if result.status_code >= 400:\n                            raise Exception(f\"HTTP {result.status_code} error for URL '{result.url}'\")\n                        original_htmls.append(result.html)\n                        cleaned = preprocess_html_for_schema(\n                            html_content=result.html,\n                            text_threshold=2000,\n                            attr_value_threshold=500,\n                            max_size=500_000\n                        )\n                        header = HTML_EXAMPLE_DELIMITER.format(index=i)","sourceCodeStart":1829,"sourceCodeEnd":1865,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/extraction_strategy.py#L1829-L1865","documentation":"Raised inside generate_schema when the single-URL fetch succeeds technically but the HTTP status is >= 400. The crawler returned a result (success=True) yet the server answered with a client/server error, so there is no usable HTML for schema inference.","triggerScenarios":"Calling generate_schema(url=X) where X returns 404, 403, 429, or 5xx. Bot walls frequently answer 403 to headless browsers; rate limiting answers 429; expired links answer 404.","commonSituations":"Generating schemas from URLs scraped from a sitemap where some entries are dead; sites that return 403 to non-browser user agents; API endpoints or PDF links mistakenly passed as pages.","solutions":["Check the URL status externally: curl -I <url> — fix or drop the URL if it is 404/410","For 403/429, add realistic headers/user-agent, delays, or a proxy via BrowserConfig/CrawlerRunConfig used by your own crawl, then pass html= instead","For multi-URL generation, pre-filter the list with HEAD requests before calling generate_schema"],"exampleFix":"// before\nschema = await JsonElementExtractionStrategy.generate_schema(\n    url=\"https://example.com/gone\")  # HTTP 404 error for URL\n\n// after\nimport httpx\nr = await httpx.AsyncClient().head(url)\nif r.status_code < 400:\n    schema = await JsonElementExtractionStrategy.generate_schema(url=url)\nelse:\n    schema = await JsonElementExtractionStrategy.generate_schema(html=local_copy_html)","handlingStrategy":"validation","validationCode":"import httpx\n\nr = await httpx.AsyncClient(follow_redirects=True).head(url)\nif r.status_code >= 400:\n    raise RuntimeError(f\"URL returns {r.status_code}; not usable for schema generation\")","typeGuard":null,"tryCatchPattern":"try:\n    schema = await JsonElementExtractionStrategy.generate_schema(url=url)\nexcept Exception as e:\n    if \"HTTP \" in str(e) and \"error for URL\" in str(e):\n        skip_or_replace_url(url)  # drop dead link, continue batch\n    raise","preventionTips":["HEAD-check URLs and drop >= 400 before calling generate_schema","Expect 403/429 from bot-walls and rate limits on headless fetches","Curate sample URLs rather than feeding raw sitemaps"],"tags":["extraction","schema-generation","http-status"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}