unclecode/crawl4ai · error · Exception

HTTP {result.status_code} error for URL '{urls[0]}'

Error message

HTTP {result.status_code} error for URL '{urls[0]}'

What it means

Raised inside generate_schema when the single-URL fetch succeeds technically but the HTTP status is >= 400. The crawler returned a result (success=True) yet the server answered with a client/server error, so there is no usable HTML for schema inference.

Source

Thrown at crawl4ai/extraction_strategy.py:1847

            from .async_configs import BrowserConfig, CrawlerRunConfig, CacheMode

            browser_config = BrowserConfig(
                headless=True,
                text_mode=True,
                light_mode=True,
            )
            crawler_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)

            # Normalize to list
            urls = [url] if isinstance(url, str) else url

            async with AsyncWebCrawler(config=browser_config) as crawler:
                if len(urls) == 1:
                    result = await crawler.arun(url=urls[0], config=crawler_config)
                    if not result.success:
                        raise Exception(f"Failed to fetch URL '{urls[0]}': {result.error_message}")
                    if result.status_code >= 400:
                        raise Exception(f"HTTP {result.status_code} error for URL '{urls[0]}'")
                    html = result.html
                    original_htmls = [result.html]
                else:
                    results = await crawler.arun_many(urls=urls, config=crawler_config)
                    html_parts = []
                    for i, result in enumerate(results, 1):
                        if not result.success:
                            raise Exception(f"Failed to fetch URL '{result.url}': {result.error_message}")
                        if result.status_code >= 400:
                            raise Exception(f"HTTP {result.status_code} error for URL '{result.url}'")
                        original_htmls.append(result.html)
                        cleaned = preprocess_html_for_schema(
                            html_content=result.html,
                            text_threshold=2000,
                            attr_value_threshold=500,
                            max_size=500_000
                        )
                        header = HTML_EXAMPLE_DELIMITER.format(index=i)

View on GitHub (pinned to 7e80152142)

Solutions

  1. Check the URL status externally: curl -I <url> — fix or drop the URL if it is 404/410
  2. For 403/429, add realistic headers/user-agent, delays, or a proxy via BrowserConfig/CrawlerRunConfig used by your own crawl, then pass html= instead
  3. For multi-URL generation, pre-filter the list with HEAD requests before calling generate_schema

Example fix

// before
schema = await JsonElementExtractionStrategy.generate_schema(
    url="https://example.com/gone")  # HTTP 404 error for URL

// after
import httpx
r = await httpx.AsyncClient().head(url)
if r.status_code < 400:
    schema = await JsonElementExtractionStrategy.generate_schema(url=url)
else:
    schema = await JsonElementExtractionStrategy.generate_schema(html=local_copy_html)
Defensive patterns

Strategy: validation

Validate before calling

import httpx

r = await httpx.AsyncClient(follow_redirects=True).head(url)
if r.status_code >= 400:
    raise RuntimeError(f"URL returns {r.status_code}; not usable for schema generation")

Try / catch

try:
    schema = await JsonElementExtractionStrategy.generate_schema(url=url)
except Exception as e:
    if "HTTP " in str(e) and "error for URL" in str(e):
        skip_or_replace_url(url)  # drop dead link, continue batch
    raise

Prevention

When it happens

Trigger: Calling generate_schema(url=X) where X returns 404, 403, 429, or 5xx. Bot walls frequently answer 403 to headless browsers; rate limiting answers 429; expired links answer 404.

Common situations: Generating schemas from URLs scraped from a sitemap where some entries are dead; sites that return 403 to non-browser user agents; API endpoints or PDF links mistakenly passed as pages.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/538c80c73e8ce53b. Report an issue: GitHub.