unclecode/crawl4ai · error · Exception

Failed to fetch URL '{urls[0]}': {result.error_message}

Error message

Failed to fetch URL '{urls[0]}': {result.error_message}

What it means

Raised inside generate_schema when the single-URL fetch path (crawler.arun on urls[0]) returns success=False. The exception message includes the target URL and result.error_message, so the underlying browser/network failure reason is preserved. Note it is a bare Exception, raised before any LLM work starts.

Source

Thrown at crawl4ai/extraction_strategy.py:1845

        if url is not None:
            from .async_webcrawler import AsyncWebCrawler
            from .async_configs import BrowserConfig, CrawlerRunConfig, CacheMode

            browser_config = BrowserConfig(
                headless=True,
                text_mode=True,
                light_mode=True,
            )
            crawler_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)

            # Normalize to list
            urls = [url] if isinstance(url, str) else url

            async with AsyncWebCrawler(config=browser_config) as crawler:
                if len(urls) == 1:
                    result = await crawler.arun(url=urls[0], config=crawler_config)
                    if not result.success:
                        raise Exception(f"Failed to fetch URL '{urls[0]}': {result.error_message}")
                    if result.status_code >= 400:
                        raise Exception(f"HTTP {result.status_code} error for URL '{urls[0]}'")
                    html = result.html
                    original_htmls = [result.html]
                else:
                    results = await crawler.arun_many(urls=urls, config=crawler_config)
                    html_parts = []
                    for i, result in enumerate(results, 1):
                        if not result.success:
                            raise Exception(f"Failed to fetch URL '{result.url}': {result.error_message}")
                        if result.status_code >= 400:
                            raise Exception(f"HTTP {result.status_code} error for URL '{result.url}'")
                        original_htmls.append(result.html)
                        cleaned = preprocess_html_for_schema(
                            html_content=result.html,
                            text_threshold=2000,
                            attr_value_threshold=500,
                            max_size=500_000

View on GitHub (pinned to 7e80152142)

Solutions

  1. Verify the URL opens in a browser and returns 200 (curl -I <url>)
  2. Crawl it manually first to see the full error: result = await AsyncWebCrawler().arun(url, config=CrawlerRunConfig(...)); print(result.error_message)
  3. Add anti-bot settings (headers, wait_for, proxy) to the crawl config or fetch the HTML yourself and pass html= instead of url=
  4. If the site blocks headless browsers, save the page HTML and call generate_schema(html=saved_html)

Example fix

// before
schema = await JsonElementExtractionStrategy.generate_schema(
    url="https://example.com/protected")  # Failed to fetch URL

// after
# fetch HTML in a real browser / with anti-bot config, then pass raw HTML
html = open("page.html").read()
schema = await JsonElementExtractionStrategy.generate_schema(html=html)
Defensive patterns

Strategy: validation

Validate before calling

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

probe = await AsyncWebCrawler().arun(url, config=CrawlerRunConfig(cache_mode=CacheMode.BYPASS))
if not probe.success or probe.status_code >= 400:
    raise RuntimeError(f"URL unusable: {probe.error_message}")
schema = await JsonElementExtractionStrategy.generate_schema(url=url)

Try / catch

try:
    schema = await JsonElementExtractionStrategy.generate_schema(url=url)
except Exception as e:
    if "Failed to fetch URL" in str(e):
        # fall back to a locally saved copy of the page
        schema = await JsonElementExtractionStrategy.generate_schema(html=local_html)
    else:
        raise

Prevention

When it happens

Trigger: Calling generate_schema(url=X) where X is unreachable, DNS fails, TLS breaks, the page times out, or the browser cannot render it (bot blocking, 403 challenges) so AsyncWebCrawler marks the result failed.

Common situations: Schema generation against sites behind Cloudflare/bot protection; unreachable intranet URLs from the developer machine; typo'd domains; headless browser blocked by the target.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/886754f7847ed32d. Report an issue: GitHub.