unclecode/crawl4ai · error · HTTPException

result.error_message

Error message

result.error_message

What it means

In the Docker server's QA endpoint (deploy/docker/api.py:151), after crawler.arun(url) the result is checked and on failure an HTTP 500 is returned whose detail is result.error_message — the crawl4ai CrawlResult's own error text (network timeout, DNS failure, page crash, navigation error, etc.). So this 500 is a crawl-level failure surfaced over HTTP, not an API-internal crash.

Source

Thrown at deploy/docker/api.py:151

        validate_url_destination(url)
        # Extract base URL by finding last '?q=' occurrence
        last_q_index = url.rfind('?q=')
        if last_q_index != -1:
            url = url[:last_q_index]

        # Get markdown content (use default config)
        from utils import load_config
        cfg = load_config()
        browser_cfg = BrowserConfig(
            extra_args=cfg["crawler"]["browser"].get("extra_args", []),
            **cfg["crawler"]["browser"].get("kwargs", {}),
        )
        from egress_broker import enforce_egress
        enforce_egress(browser_cfg)
        crawler = await get_crawler(browser_cfg)
        result = await crawler.arun(url)
        if not result.success:
            raise HTTPException(
                status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
                detail=result.error_message
            )
        content = result.markdown.fit_markdown or result.markdown.raw_markdown

        # Create prompt and get LLM response
        prompt = f"""Use the following content as context to answer the question.
    Content:
    {content}

    Question: {query}

    Answer:"""

        # Provider by name only; base_url/api_token are server-derived. A
        # request-supplied base_url is ignored (it was the key-exfil vector).
        from llm_broker import resolve_llm
        llm = resolve_llm(config, provider)

View on GitHub (pinned to 7e80152142)

Solutions

1. Read the detail field — it is the crawler's error message (e.g. 'net::ERR_NAME_NOT_RESOLVED') and pinpoints the cause.
2. Verify the URL is reachable (curl) from inside the container; fix DNS/proxy/egress rules (enforce_egress may also block).
3. For bot-blocked sites, adjust BrowserConfig (headers, proxy) or run playwright install in the image.
4. Retry transient network failures with backoff; treat persistent failures as a bad URL, not a server bug.

Example fix

# before
r = requests.post(f"{base}/q", json={"url": url, "query": q})
r.raise_for_status()  # opaque 500

# after
r = requests.post(f"{base}/q", json={"url": url, "query": q})
if r.status_code == 500:
    logging.error("crawl failed: %s", r.json().get("detail"))  # actionable crawler error
    # fix URL / egress / browser deps based on the message

from urllib.parse import urlparse

def url_likely_crawlable(url: str) -> bool:
    p = urlparse(url if "//" in url else "https://" + url)
    return p.scheme in ("http", "https") and bool(p.netloc) and "." in p.netloc

r = await client.post("/q", json=body)
if r.status_code == 500:
    detail = r.json().get("detail", "")
    if "ERR_NAME_NOT_RESOLVED" in detail or "Timeout" in detail:
        await asyncio.sleep(2)  # transient network — retry once
        r = await client.post("/q", json=body)
    else:
        raise RuntimeError(f"crawl failed: {detail}")

When it happens

Trigger: POST to the QA/question endpoint with a URL that fails to load: unreachable host, TLS error, 403/anti-bot block, Playwright browser crash, or the crawl strategy raising — CrawlResult.success becomes False and error_message is propagated as the 500 detail.

Common situations: Passing unreachable or typo'd URLs (api.py normalizes missing schemes by prepending https://, turning typos into DNS failures); crawling bot-protected sites; container missing browser dependencies so every arun fails.

Related errors

• Crawl request failed: {results['results'][0]['error_message'
• Error evaluating condition: ${{error.message}}
• Invalid hook type: {hook_type}
• Crawl failed: {result_data.get('msg', 'Unknown error')}
• Too many concurrent jobs for this caller

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/16465cb6c8ec62a8. Report an issue: GitHub.