unclecode/crawl4ai · error · HTTPException

Crawl request failed: {results['results'][0]['error_message'

Error message

Crawl request failed: {results['results'][0]['error_message']}

What it means

Raised as HTTP 500 by POST /crawl when every URL in the batch failed. handle_crawl_request already caught each individual failure and recorded it in results['results'][i]['error_message']; the endpoint only escalates to a 500 when ALL results have success=False, surfacing the first error message as the detail.

Source

Thrown at deploy/docker/server.py:911

    # Prepare hooks config if provided
    hooks_config = None
    if crawl_request.hooks:
        hooks_config = {
            'hooks': crawl_request.hooks.hooks,
            'timeout': crawl_request.hooks.timeout
        }
    
    results = await handle_crawl_request(
        urls=crawl_request.urls,
        browser_config=crawl_request.browser_config,
        crawler_config=crawl_request.crawler_config,
        config=config,
        hooks_config=hooks_config,
        crawler_configs=crawl_request.crawler_configs,
    )
    # check if all of the results are not successful
    if all(not result["success"] for result in results["results"]):
        raise HTTPException(500, f"Crawl request failed: {results['results'][0]['error_message']}")
    return JSONResponse(results)


@app.post("/crawl/stream")
@limiter.limit(config["rate_limiting"]["default_limit"])
async def crawl_stream(
    request: Request,
    crawl_request: CrawlRequestWithHooks,
    _td: Dict = Depends(token_dep),
):
    if not crawl_request.urls:
        raise HTTPException(400, "At least one URL required")
    if crawl_request.hooks and not HOOKS_ENABLED:
        raise HTTPException(403, "Hooks are disabled. Set CRAWL4AI_HOOKS_ENABLED=true to enable.")

    return await stream_process(crawl_request=crawl_request)

async def stream_process(crawl_request: CrawlRequestWithHooks):

View on GitHub (pinned to 7e80152142)

Solutions

1. Crawl one URL with the same browser_config/crawler_config to isolate whether it is config or environment
2. Check container egress: docker run --rm the image and curl one of the target URLs; fix DNS/proxy/firewall if blocked
3. Loosen crawler_config timeouts (page_timeout, delay) and disable JS rendering (js_code='') to see if the pages load at all
4. Inspect the per-result error_message payloads of a 200 response from a mixed batch - they name the real per-URL cause
5. If failures are intermittent, wrap the API call in retry with backoff rather than tuning config

Example fix

# before
resp = requests.post(f"{SERVER}/crawl", json={"urls": urls}, headers=AUTH)
resp.raise_for_status()

# after
resp = requests.post(f"{SERVER}/crawl", json={"urls": urls}, headers=AUTH)
if resp.status_code == 500:
    print("all failed, first error:", resp.json()["detail"])
    resp = requests.post(f"{SERVER}/crawl", json={"urls": urls[:1]}, headers=AUTH)  # bisect the failing URL

# pre-flight: confirm target reachability before batching
import requests
ok = [u for u in urls if requests.head(u, timeout=5, allow_redirects=True).status_code < 500]

try:
    resp = requests.post(f"{S}/crawl", json={"urls": urls}, headers=AUTH, timeout=300)
    resp.raise_for_status()
except requests.HTTPError as e:
    if e.response.status_code == 500:
        detail = e.response.json().get("detail", "")
        # bisect: retry per-URL to salvage partial results
        results = [requests.post(f"{S}/crawl", json={"urls": [u]}, headers=AUTH) for u in urls]

When it happens

Trigger: POST /crawl (or /crawl/db-sync etc. reusing handle_crawl_request) with token auth, where every requested URL fails during the crawl: unreachable hosts, DNS errors, timeouts, JS-rendered pages that never load, or all URLs rejected by SSRF/robots checks. Partial success (>=1 result success=true) returns 200 with per-result errors instead.

Common situations: Bad/expired target URLs, container has no network egress, Playwright browser crash killing all crawls, an aggressive crawler_config (tiny page timeout) failing every page, or a firewall blocking all target hosts. Also seen when a shared proxy env var routes all requests to a dead proxy.

Related errors

• Too many concurrent jobs for this caller
• result.error_message
• Error evaluating condition: ${{error.message}}
• Invalid hook type: {hook_type}
• Invalid CSS selector , No elements found for CSS selector: {

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/47e788af34b1587d. Report an issue: GitHub.