unclecode/crawl4ai · error · Exception

Failed to fetch URL '{result.url}': {result.error_message}

Error message

Failed to fetch URL '{result.url}': {result.error_message}

What it means

Multi-URL variant of the fetch failure in generate_schema: during crawler.arun_many over the URL list, a result comes back with success=False, and the loop aborts immediately with the failing URL and its error_message. One bad URL kills the whole batch, so its position in the list matters.

Source

Thrown at crawl4ai/extraction_strategy.py:1855

            # Normalize to list
            urls = [url] if isinstance(url, str) else url

            async with AsyncWebCrawler(config=browser_config) as crawler:
                if len(urls) == 1:
                    result = await crawler.arun(url=urls[0], config=crawler_config)
                    if not result.success:
                        raise Exception(f"Failed to fetch URL '{urls[0]}': {result.error_message}")
                    if result.status_code >= 400:
                        raise Exception(f"HTTP {result.status_code} error for URL '{urls[0]}'")
                    html = result.html
                    original_htmls = [result.html]
                else:
                    results = await crawler.arun_many(urls=urls, config=crawler_config)
                    html_parts = []
                    for i, result in enumerate(results, 1):
                        if not result.success:
                            raise Exception(f"Failed to fetch URL '{result.url}': {result.error_message}")
                        if result.status_code >= 400:
                            raise Exception(f"HTTP {result.status_code} error for URL '{result.url}'")
                        original_htmls.append(result.html)
                        cleaned = preprocess_html_for_schema(
                            html_content=result.html,
                            text_threshold=2000,
                            attr_value_threshold=500,
                            max_size=500_000
                        )
                        header = HTML_EXAMPLE_DELIMITER.format(index=i)
                        html_parts.append(f"{header}\n{cleaned}")
                    html = "\n\n".join(html_parts)
        else:
            original_htmls = [html]

        # Preprocess HTML for schema generation (skip if already preprocessed from multiple URLs)
        if url is None or isinstance(url, str):
            html = preprocess_html_for_schema(

View on GitHub (pinned to 7e80152142)

Solutions

  1. Identify the failing URL from the message and remove/fix it, then retry the batch
  2. Pre-validate all URLs with lightweight HEAD/GET checks before generating the schema
  3. Fall back to single-URL generation for the healthy URLs, or pass pre-fetched html strings

Example fix

// before
schema = await JsonElementExtractionStrategy.generate_schema(
    url=[u1, u2, broken_u3])  # Failed to fetch URL

// after
ok_urls = [u for u in urls if await reachable(u)]
schema = await JsonElementExtractionStrategy.generate_schema(url=ok_urls)
Defensive patterns

Strategy: validation

Validate before calling

import httpx

async def filter_reachable(urls):
    async with httpx.AsyncClient(timeout=10) as hc:
        ok = []
        for u in urls:
            try:
                r = await hc.head(u, follow_redirects=True)
                if r.status_code < 400:
                    ok.append(u)
            except httpx.HTTPError:
                continue
        return ok

urls = await filter_reachable(urls)

Try / catch

try:
    schema = await JsonElementExtractionStrategy.generate_schema(url=urls)
except Exception as e:
    if "Failed to fetch URL" in str(e):
        bad = extract_url_from_error(str(e))
        urls = [u for u in urls if u != bad]
        schema = await JsonElementExtractionStrategy.generate_schema(url=urls)
    raise

Prevention

When it happens

Trigger: Calling generate_schema(url=[u1, u2, u3]) where any single URL fails at the network/browser level (unreachable, DNS, TLS, timeout, bot-block). The first failing result raises.

Common situations: Feeding a list of product URLs where one has gone offline; mixed lists containing unreachable intranet hosts; large lists increasing the chance one URL times out under concurrency.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/76160b9a9dbf2344. Report an issue: GitHub.