unclecode/crawl4ai · error · HTTPException

Crawl failed

Error message

Crawl failed

What it means

A 500 from the HTML-preprocessing endpoint: crawler.arun() completed but results[0].success was falsy, and results[0].error_message was empty, so the generic 'Crawl failed' detail was used. It means the browser crawler returned a failure result (navigation error, timeout, blocked page) without a specific message.

Source

Thrown at deploy/docker/server.py:620

@limiter.limit(config["rate_limiting"]["default_limit"])
@mcp_tool("html")
async def generate_html(
    request: Request,
    body: HTMLRequest,
    _td: Dict = Depends(token_dep),
):
    """
    Crawls the URL, preprocesses the raw HTML for schema extraction, and returns the processed HTML.
    Use when you need sanitized HTML structures for building schemas or further processing.
    """
    validate_url_scheme(body.url, allow_raw=True)
    cfg = CrawlerRunConfig()
    crawler = None
    try:
        crawler = await get_crawler(get_default_browser_config())
        results = await crawler.arun(url=body.url, config=cfg)
        if not results[0].success:
            raise HTTPException(500, detail=results[0].error_message or "Crawl failed")

        raw_html = results[0].html
        from crawl4ai.utils import preprocess_html_for_schema
        processed_html = preprocess_html_for_schema(raw_html)
        return JSONResponse({"html": processed_html, "url": body.url, "success": True})
    except Exception as e:
        raise HTTPException(500, detail=str(e))
    finally:
        if crawler:
            await release_crawler(crawler)

# ── artifact store helpers ───────────────────────────────────
def _store_artifact(kind: str, data: bytes) -> dict:
    """Write to the sandboxed store; map quota/size errors to HTTP codes."""
    from artifacts import write_artifact, ArtifactTooLarge, QuotaExceeded
    try:
        meta = write_artifact(kind, data)
    except ArtifactTooLarge:

View on GitHub (pinned to 7e80152142)

Solutions

1. Retry once — many crawl failures are transient network/timing issues.
2. Verify the URL loads in a normal browser from the same network/host as the container.
3. If the site blocks bots, add browser-humanizing config (headers, wait_for) via CrawlerRunConfig.
4. Check container logs for Playwright/browser errors; ensure the browser image/dependencies are installed and memory is sufficient.

import requests

def url_reachable(url: str) -> bool:
    try:
        return requests.head(url, timeout=10, allow_redirects=True).status_code < 500
    except requests.RequestException:
        return False

for attempt in range(3):
    resp = requests.post(f'{BASE}/preprocess', json={'url': url}, headers=hdrs)
    if resp.status_code == 500 and 'Crawl failed' in resp.text:
        time.sleep(2 ** attempt)
        continue
    resp.raise_for_status()
    break

When it happens

Trigger: POST to the endpoint with a URL that fails to load: DNS failure, TLS error, 403/robot-blocked page, page crash, or a crawler timeout. validate_url_scheme passed (raw: allowed) but the actual navigation failed.

Common situations: Target site blocks headless browsers (Cloudflare, 403s); URL behind auth or geo-restriction; headless browser resource exhaustion in the container; transient network issues in Docker deployments.

Related errors

• Error evaluating condition: ${{error.message}}
• Failed on navigating ACS-GOTO: {str(e)}
• Invalid hook type: {hook_type}
• Body element is hidden: {visibility_info}
• Failed to extract HTML content: {str(e)}

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/640b813376ad408d. Report an issue: GitHub.