unclecode/crawl4ai · error · Exception

HTTP {result.status_code} error for URL '{result.url}'

Error message

HTTP {result.status_code} error for URL '{result.url}'

What it means

Multi-URL variant of the HTTP-status check in generate_schema: during arun_many, a result has success=True but status_code >= 400, and the loop aborts with the failing URL and status. Like its single-URL twin, any one erroring URL stops schema generation for the entire list.

Source

Thrown at crawl4ai/extraction_strategy.py:1857

            urls = [url] if isinstance(url, str) else url

            async with AsyncWebCrawler(config=browser_config) as crawler:
                if len(urls) == 1:
                    result = await crawler.arun(url=urls[0], config=crawler_config)
                    if not result.success:
                        raise Exception(f"Failed to fetch URL '{urls[0]}': {result.error_message}")
                    if result.status_code >= 400:
                        raise Exception(f"HTTP {result.status_code} error for URL '{urls[0]}'")
                    html = result.html
                    original_htmls = [result.html]
                else:
                    results = await crawler.arun_many(urls=urls, config=crawler_config)
                    html_parts = []
                    for i, result in enumerate(results, 1):
                        if not result.success:
                            raise Exception(f"Failed to fetch URL '{result.url}': {result.error_message}")
                        if result.status_code >= 400:
                            raise Exception(f"HTTP {result.status_code} error for URL '{result.url}'")
                        original_htmls.append(result.html)
                        cleaned = preprocess_html_for_schema(
                            html_content=result.html,
                            text_threshold=2000,
                            attr_value_threshold=500,
                            max_size=500_000
                        )
                        header = HTML_EXAMPLE_DELIMITER.format(index=i)
                        html_parts.append(f"{header}\n{cleaned}")
                    html = "\n\n".join(html_parts)
        else:
            original_htmls = [html]

        # Preprocess HTML for schema generation (skip if already preprocessed from multiple URLs)
        if url is None or isinstance(url, str):
            html = preprocess_html_for_schema(
                html_content=html,
                text_threshold=2000,

View on GitHub (pinned to 7e80152142)

Solutions

  1. Pre-screen every URL's status code with HEAD requests and drop >= 400 before the call
  2. Reduce concurrency or add delays so the target does not answer 429
  3. Pass locally saved HTML for problematic pages via html= / curated URL lists instead

Example fix

// before
schema = await JsonElementExtractionStrategy.generate_schema(url=sitemap_urls)
# HTTP 403 error for URL '...'

// after
import httpx
async with httpx.AsyncClient(follow_redirects=True) as hc:
    ok = [u for u in sitemap_urls
          if (await hc.head(u)).status_code < 400]
schema = await JsonElementExtractionStrategy.generate_schema(url=ok)
Defensive patterns

Strategy: validation

Validate before calling

import httpx

async def status_ok(u: str) -> bool:
    try:
        r = await httpx.AsyncClient(follow_redirects=True).head(u, timeout=10)
        return r.status_code < 400
    except httpx.HTTPError:
        return False

urls = [u for u in urls if await status_ok(u)]

Try / catch

try:
    schema = await JsonElementExtractionStrategy.generate_schema(url=urls)
except Exception as e:
    if "error for URL" in str(e):  # HTTP nnn branch
        bad = extract_url_from_error(str(e))
        schema = await JsonElementExtractionStrategy.generate_schema(
            url=[u for u in urls if u != bad])
    raise

Prevention

When it happens

Trigger: Calling generate_schema(url=[...]) where any URL answers 403/404/429/5xx to the headless browser; expired sitemap entries; endpoints that reject headless clients.

Common situations: Bulk schema generation from sitemaps or category-page lists containing dead links; rate-limited targets returning 429 under the concurrency of arun_many; sites returning 403 without browser-like headers.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/2df7d9d1a4ca7643. Report an issue: GitHub.