{"record":{"id":"886754f7847ed32d","repo":"unclecode/crawl4ai","slug":"failed-to-fetch-url-urls-0-result-error-mes","errorCode":null,"errorMessage":"Failed to fetch URL '{urls[0]}': {result.error_message}","messagePattern":"Failed to fetch URL '(.+?)': (.+?)","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"crawl4ai/extraction_strategy.py","lineNumber":1845,"sourceCode":"        if url is not None:\n            from .async_webcrawler import AsyncWebCrawler\n            from .async_configs import BrowserConfig, CrawlerRunConfig, CacheMode\n\n            browser_config = BrowserConfig(\n                headless=True,\n                text_mode=True,\n                light_mode=True,\n            )\n            crawler_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)\n\n            # Normalize to list\n            urls = [url] if isinstance(url, str) else url\n\n            async with AsyncWebCrawler(config=browser_config) as crawler:\n                if len(urls) == 1:\n                    result = await crawler.arun(url=urls[0], config=crawler_config)\n                    if not result.success:\n                        raise Exception(f\"Failed to fetch URL '{urls[0]}': {result.error_message}\")\n                    if result.status_code >= 400:\n                        raise Exception(f\"HTTP {result.status_code} error for URL '{urls[0]}'\")\n                    html = result.html\n                    original_htmls = [result.html]\n                else:\n                    results = await crawler.arun_many(urls=urls, config=crawler_config)\n                    html_parts = []\n                    for i, result in enumerate(results, 1):\n                        if not result.success:\n                            raise Exception(f\"Failed to fetch URL '{result.url}': {result.error_message}\")\n                        if result.status_code >= 400:\n                            raise Exception(f\"HTTP {result.status_code} error for URL '{result.url}'\")\n                        original_htmls.append(result.html)\n                        cleaned = preprocess_html_for_schema(\n                            html_content=result.html,\n                            text_threshold=2000,\n                            attr_value_threshold=500,\n                            max_size=500_000","sourceCodeStart":1827,"sourceCodeEnd":1863,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/extraction_strategy.py#L1827-L1863","documentation":"Raised inside generate_schema when the single-URL fetch path (crawler.arun on urls[0]) returns success=False. The exception message includes the target URL and result.error_message, so the underlying browser/network failure reason is preserved. Note it is a bare Exception, raised before any LLM work starts.","triggerScenarios":"Calling generate_schema(url=X) where X is unreachable, DNS fails, TLS breaks, the page times out, or the browser cannot render it (bot blocking, 403 challenges) so AsyncWebCrawler marks the result failed.","commonSituations":"Schema generation against sites behind Cloudflare/bot protection; unreachable intranet URLs from the developer machine; typo'd domains; headless browser blocked by the target.","solutions":["Verify the URL opens in a browser and returns 200 (curl -I <url>)","Crawl it manually first to see the full error: result = await AsyncWebCrawler().arun(url, config=CrawlerRunConfig(...)); print(result.error_message)","Add anti-bot settings (headers, wait_for, proxy) to the crawl config or fetch the HTML yourself and pass html= instead of url=","If the site blocks headless browsers, save the page HTML and call generate_schema(html=saved_html)"],"exampleFix":"// before\nschema = await JsonElementExtractionStrategy.generate_schema(\n    url=\"https://example.com/protected\")  # Failed to fetch URL\n\n// after\n# fetch HTML in a real browser / with anti-bot config, then pass raw HTML\nhtml = open(\"page.html\").read()\nschema = await JsonElementExtractionStrategy.generate_schema(html=html)","handlingStrategy":"validation","validationCode":"from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode\n\nprobe = await AsyncWebCrawler().arun(url, config=CrawlerRunConfig(cache_mode=CacheMode.BYPASS))\nif not probe.success or probe.status_code >= 400:\n    raise RuntimeError(f\"URL unusable: {probe.error_message}\")\nschema = await JsonElementExtractionStrategy.generate_schema(url=url)","typeGuard":null,"tryCatchPattern":"try:\n    schema = await JsonElementExtractionStrategy.generate_schema(url=url)\nexcept Exception as e:\n    if \"Failed to fetch URL\" in str(e):\n        # fall back to a locally saved copy of the page\n        schema = await JsonElementExtractionStrategy.generate_schema(html=local_html)\n    else:\n        raise","preventionTips":["Crawl the URL once yourself to confirm it renders in a headless browser","For bot-protected sites, fetch HTML manually and pass html=","Check DNS/URL validity before schema generation"],"tags":["extraction","schema-generation","fetch-failure","network"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}