{"record":{"id":"640b813376ad408d","repo":"unclecode/crawl4ai","slug":"crawl-failed","errorCode":null,"errorMessage":"Crawl failed","messagePattern":"Crawl failed","errorType":"http","errorClass":"HTTPException","httpStatus":500,"severity":"error","filePath":"deploy/docker/server.py","lineNumber":620,"sourceCode":"@limiter.limit(config[\"rate_limiting\"][\"default_limit\"])\n@mcp_tool(\"html\")\nasync def generate_html(\n    request: Request,\n    body: HTMLRequest,\n    _td: Dict = Depends(token_dep),\n):\n    \"\"\"\n    Crawls the URL, preprocesses the raw HTML for schema extraction, and returns the processed HTML.\n    Use when you need sanitized HTML structures for building schemas or further processing.\n    \"\"\"\n    validate_url_scheme(body.url, allow_raw=True)\n    cfg = CrawlerRunConfig()\n    crawler = None\n    try:\n        crawler = await get_crawler(get_default_browser_config())\n        results = await crawler.arun(url=body.url, config=cfg)\n        if not results[0].success:\n            raise HTTPException(500, detail=results[0].error_message or \"Crawl failed\")\n\n        raw_html = results[0].html\n        from crawl4ai.utils import preprocess_html_for_schema\n        processed_html = preprocess_html_for_schema(raw_html)\n        return JSONResponse({\"html\": processed_html, \"url\": body.url, \"success\": True})\n    except Exception as e:\n        raise HTTPException(500, detail=str(e))\n    finally:\n        if crawler:\n            await release_crawler(crawler)\n\n# ── artifact store helpers ───────────────────────────────────\ndef _store_artifact(kind: str, data: bytes) -> dict:\n    \"\"\"Write to the sandboxed store; map quota/size errors to HTTP codes.\"\"\"\n    from artifacts import write_artifact, ArtifactTooLarge, QuotaExceeded\n    try:\n        meta = write_artifact(kind, data)\n    except ArtifactTooLarge:","sourceCodeStart":602,"sourceCodeEnd":638,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/deploy/docker/server.py#L602-L638","documentation":"A 500 from the HTML-preprocessing endpoint: crawler.arun() completed but results[0].success was falsy, and results[0].error_message was empty, so the generic 'Crawl failed' detail was used. It means the browser crawler returned a failure result (navigation error, timeout, blocked page) without a specific message.","triggerScenarios":"POST to the endpoint with a URL that fails to load: DNS failure, TLS error, 403/robot-blocked page, page crash, or a crawler timeout. validate_url_scheme passed (raw: allowed) but the actual navigation failed.","commonSituations":"Target site blocks headless browsers (Cloudflare, 403s); URL behind auth or geo-restriction; headless browser resource exhaustion in the container; transient network issues in Docker deployments.","solutions":["Retry once — many crawl failures are transient network/timing issues.","Verify the URL loads in a normal browser from the same network/host as the container.","If the site blocks bots, add browser-humanizing config (headers, wait_for) via CrawlerRunConfig.","Check container logs for Playwright/browser errors; ensure the browser image/dependencies are installed and memory is sufficient."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"import requests\n\ndef url_reachable(url: str) -> bool:\n    try:\n        return requests.head(url, timeout=10, allow_redirects=True).status_code < 500\n    except requests.RequestException:\n        return False","typeGuard":null,"tryCatchPattern":"for attempt in range(3):\n    resp = requests.post(f'{BASE}/preprocess', json={'url': url}, headers=hdrs)\n    if resp.status_code == 500 and 'Crawl failed' in resp.text:\n        time.sleep(2 ** attempt)\n        continue\n    resp.raise_for_status()\n    break","preventionTips":["Pre-check reachability with a cheap HEAD request before paying for a browser crawl.","Retry with exponential backoff; many crawl failures are transient.","Watch container memory/browser health — chronic 'Crawl failed' with empty error_message often means browser starvation."],"tags":["crawl","network","http-500","playwright"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}