{"record":{"id":"47e788af34b1587d","repo":"unclecode/crawl4ai","slug":"crawl-request-failed-results-results-0-erro","errorCode":null,"errorMessage":"Crawl request failed: {results['results'][0]['error_message']}","messagePattern":"Crawl request failed: (.+?)","errorType":"http","errorClass":"HTTPException","httpStatus":500,"severity":"error","filePath":"deploy/docker/server.py","lineNumber":911,"sourceCode":"    # Prepare hooks config if provided\n    hooks_config = None\n    if crawl_request.hooks:\n        hooks_config = {\n            'hooks': crawl_request.hooks.hooks,\n            'timeout': crawl_request.hooks.timeout\n        }\n    \n    results = await handle_crawl_request(\n        urls=crawl_request.urls,\n        browser_config=crawl_request.browser_config,\n        crawler_config=crawl_request.crawler_config,\n        config=config,\n        hooks_config=hooks_config,\n        crawler_configs=crawl_request.crawler_configs,\n    )\n    # check if all of the results are not successful\n    if all(not result[\"success\"] for result in results[\"results\"]):\n        raise HTTPException(500, f\"Crawl request failed: {results['results'][0]['error_message']}\")\n    return JSONResponse(results)\n\n\n@app.post(\"/crawl/stream\")\n@limiter.limit(config[\"rate_limiting\"][\"default_limit\"])\nasync def crawl_stream(\n    request: Request,\n    crawl_request: CrawlRequestWithHooks,\n    _td: Dict = Depends(token_dep),\n):\n    if not crawl_request.urls:\n        raise HTTPException(400, \"At least one URL required\")\n    if crawl_request.hooks and not HOOKS_ENABLED:\n        raise HTTPException(403, \"Hooks are disabled. Set CRAWL4AI_HOOKS_ENABLED=true to enable.\")\n\n    return await stream_process(crawl_request=crawl_request)\n\nasync def stream_process(crawl_request: CrawlRequestWithHooks):","sourceCodeStart":893,"sourceCodeEnd":929,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/deploy/docker/server.py#L893-L929","documentation":"Raised as HTTP 500 by POST /crawl when every URL in the batch failed. handle_crawl_request already caught each individual failure and recorded it in results['results'][i]['error_message']; the endpoint only escalates to a 500 when ALL results have success=False, surfacing the first error message as the detail.","triggerScenarios":"POST /crawl (or /crawl/db-sync etc. reusing handle_crawl_request) with token auth, where every requested URL fails during the crawl: unreachable hosts, DNS errors, timeouts, JS-rendered pages that never load, or all URLs rejected by SSRF/robots checks. Partial success (>=1 result success=true) returns 200 with per-result errors instead.","commonSituations":"Bad/expired target URLs, container has no network egress, Playwright browser crash killing all crawls, an aggressive crawler_config (tiny page timeout) failing every page, or a firewall blocking all target hosts. Also seen when a shared proxy env var routes all requests to a dead proxy.","solutions":["Crawl one URL with the same browser_config/crawler_config to isolate whether it is config or environment","Check container egress: docker run --rm the image and curl one of the target URLs; fix DNS/proxy/firewall if blocked","Loosen crawler_config timeouts (page_timeout, delay) and disable JS rendering (js_code='') to see if the pages load at all","Inspect the per-result error_message payloads of a 200 response from a mixed batch - they name the real per-URL cause","If failures are intermittent, wrap the API call in retry with backoff rather than tuning config"],"exampleFix":"# before\nresp = requests.post(f\"{SERVER}/crawl\", json={\"urls\": urls}, headers=AUTH)\nresp.raise_for_status()\n\n# after\nresp = requests.post(f\"{SERVER}/crawl\", json={\"urls\": urls}, headers=AUTH)\nif resp.status_code == 500:\n    print(\"all failed, first error:\", resp.json()[\"detail\"])\n    resp = requests.post(f\"{SERVER}/crawl\", json={\"urls\": urls[:1]}, headers=AUTH)  # bisect the failing URL","handlingStrategy":"retry","validationCode":"# pre-flight: confirm target reachability before batching\nimport requests\nok = [u for u in urls if requests.head(u, timeout=5, allow_redirects=True).status_code < 500]","typeGuard":null,"tryCatchPattern":"try:\n    resp = requests.post(f\"{S}/crawl\", json={\"urls\": urls}, headers=AUTH, timeout=300)\n    resp.raise_for_status()\nexcept requests.HTTPError as e:\n    if e.response.status_code == 500:\n        detail = e.response.json().get(\"detail\", \"\")\n        # bisect: retry per-URL to salvage partial results\n        results = [requests.post(f\"{S}/crawl\", json={\"urls\": [u]}, headers=AUTH) for u in urls]","preventionTips":["Batch heterogeneous URLs in small groups so one bad host cannot fail an entire request","Monitor the per-result success flags in 200 responses to catch rising failure rates early","Run a one-URL canary crawl against a new deployment before sending production batches"],"tags":["crawl4ai","server","http-500","batch-crawl"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}