{"record":{"id":"76160b9a9dbf2344","repo":"unclecode/crawl4ai","slug":"failed-to-fetch-url-result-url-result-error","errorCode":null,"errorMessage":"Failed to fetch URL '{result.url}': {result.error_message}","messagePattern":"Failed to fetch URL '(.+?)': (.+?)","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"crawl4ai/extraction_strategy.py","lineNumber":1855,"sourceCode":"\n            # Normalize to list\n            urls = [url] if isinstance(url, str) else url\n\n            async with AsyncWebCrawler(config=browser_config) as crawler:\n                if len(urls) == 1:\n                    result = await crawler.arun(url=urls[0], config=crawler_config)\n                    if not result.success:\n                        raise Exception(f\"Failed to fetch URL '{urls[0]}': {result.error_message}\")\n                    if result.status_code >= 400:\n                        raise Exception(f\"HTTP {result.status_code} error for URL '{urls[0]}'\")\n                    html = result.html\n                    original_htmls = [result.html]\n                else:\n                    results = await crawler.arun_many(urls=urls, config=crawler_config)\n                    html_parts = []\n                    for i, result in enumerate(results, 1):\n                        if not result.success:\n                            raise Exception(f\"Failed to fetch URL '{result.url}': {result.error_message}\")\n                        if result.status_code >= 400:\n                            raise Exception(f\"HTTP {result.status_code} error for URL '{result.url}'\")\n                        original_htmls.append(result.html)\n                        cleaned = preprocess_html_for_schema(\n                            html_content=result.html,\n                            text_threshold=2000,\n                            attr_value_threshold=500,\n                            max_size=500_000\n                        )\n                        header = HTML_EXAMPLE_DELIMITER.format(index=i)\n                        html_parts.append(f\"{header}\\n{cleaned}\")\n                    html = \"\\n\\n\".join(html_parts)\n        else:\n            original_htmls = [html]\n\n        # Preprocess HTML for schema generation (skip if already preprocessed from multiple URLs)\n        if url is None or isinstance(url, str):\n            html = preprocess_html_for_schema(","sourceCodeStart":1837,"sourceCodeEnd":1873,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/extraction_strategy.py#L1837-L1873","documentation":"Multi-URL variant of the fetch failure in generate_schema: during crawler.arun_many over the URL list, a result comes back with success=False, and the loop aborts immediately with the failing URL and its error_message. One bad URL kills the whole batch, so its position in the list matters.","triggerScenarios":"Calling generate_schema(url=[u1, u2, u3]) where any single URL fails at the network/browser level (unreachable, DNS, TLS, timeout, bot-block). The first failing result raises.","commonSituations":"Feeding a list of product URLs where one has gone offline; mixed lists containing unreachable intranet hosts; large lists increasing the chance one URL times out under concurrency.","solutions":["Identify the failing URL from the message and remove/fix it, then retry the batch","Pre-validate all URLs with lightweight HEAD/GET checks before generating the schema","Fall back to single-URL generation for the healthy URLs, or pass pre-fetched html strings"],"exampleFix":"// before\nschema = await JsonElementExtractionStrategy.generate_schema(\n    url=[u1, u2, broken_u3])  # Failed to fetch URL\n\n// after\nok_urls = [u for u in urls if await reachable(u)]\nschema = await JsonElementExtractionStrategy.generate_schema(url=ok_urls)","handlingStrategy":"validation","validationCode":"import httpx\n\nasync def filter_reachable(urls):\n    async with httpx.AsyncClient(timeout=10) as hc:\n        ok = []\n        for u in urls:\n            try:\n                r = await hc.head(u, follow_redirects=True)\n                if r.status_code < 400:\n                    ok.append(u)\n            except httpx.HTTPError:\n                continue\n        return ok\n\nurls = await filter_reachable(urls)","typeGuard":null,"tryCatchPattern":"try:\n    schema = await JsonElementExtractionStrategy.generate_schema(url=urls)\nexcept Exception as e:\n    if \"Failed to fetch URL\" in str(e):\n        bad = extract_url_from_error(str(e))\n        urls = [u for u in urls if u != bad]\n        schema = await JsonElementExtractionStrategy.generate_schema(url=urls)\n    raise","preventionTips":["Pre-screen every URL in a batch — one failure aborts all","Parse the failing URL from the message and retry without it","Keep batches small so one bad URL costs little"],"tags":["extraction","schema-generation","fetch-failure","batch"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}