{"record":{"id":"16465cb6c8ec62a8","repo":"unclecode/crawl4ai","slug":"result-error-message","errorCode":null,"errorMessage":"result.error_message","messagePattern":"result\\.error_message","errorType":"http","errorClass":"HTTPException","httpStatus":500,"severity":"error","filePath":"deploy/docker/api.py","lineNumber":151,"sourceCode":"        validate_url_destination(url)\n        # Extract base URL by finding last '?q=' occurrence\n        last_q_index = url.rfind('?q=')\n        if last_q_index != -1:\n            url = url[:last_q_index]\n\n        # Get markdown content (use default config)\n        from utils import load_config\n        cfg = load_config()\n        browser_cfg = BrowserConfig(\n            extra_args=cfg[\"crawler\"][\"browser\"].get(\"extra_args\", []),\n            **cfg[\"crawler\"][\"browser\"].get(\"kwargs\", {}),\n        )\n        from egress_broker import enforce_egress\n        enforce_egress(browser_cfg)\n        crawler = await get_crawler(browser_cfg)\n        result = await crawler.arun(url)\n        if not result.success:\n            raise HTTPException(\n                status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,\n                detail=result.error_message\n            )\n        content = result.markdown.fit_markdown or result.markdown.raw_markdown\n\n        # Create prompt and get LLM response\n        prompt = f\"\"\"Use the following content as context to answer the question.\n    Content:\n    {content}\n\n    Question: {query}\n\n    Answer:\"\"\"\n\n        # Provider by name only; base_url/api_token are server-derived. A\n        # request-supplied base_url is ignored (it was the key-exfil vector).\n        from llm_broker import resolve_llm\n        llm = resolve_llm(config, provider)","sourceCodeStart":133,"sourceCodeEnd":169,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/deploy/docker/api.py#L133-L169","documentation":"In the Docker server's QA endpoint (deploy/docker/api.py:151), after crawler.arun(url) the result is checked and on failure an HTTP 500 is returned whose detail is result.error_message — the crawl4ai CrawlResult's own error text (network timeout, DNS failure, page crash, navigation error, etc.). So this 500 is a crawl-level failure surfaced over HTTP, not an API-internal crash.","triggerScenarios":"POST to the QA/question endpoint with a URL that fails to load: unreachable host, TLS error, 403/anti-bot block, Playwright browser crash, or the crawl strategy raising — CrawlResult.success becomes False and error_message is propagated as the 500 detail.","commonSituations":"Passing unreachable or typo'd URLs (api.py normalizes missing schemes by prepending https://, turning typos into DNS failures); crawling bot-protected sites; container missing browser dependencies so every arun fails.","solutions":["Read the detail field — it is the crawler's error message (e.g. 'net::ERR_NAME_NOT_RESOLVED') and pinpoints the cause.","Verify the URL is reachable (curl) from inside the container; fix DNS/proxy/egress rules (enforce_egress may also block).","For bot-blocked sites, adjust BrowserConfig (headers, proxy) or run playwright install in the image.","Retry transient network failures with backoff; treat persistent failures as a bad URL, not a server bug."],"exampleFix":"# before\nr = requests.post(f\"{base}/q\", json={\"url\": url, \"query\": q})\nr.raise_for_status()  # opaque 500\n\n# after\nr = requests.post(f\"{base}/q\", json={\"url\": url, \"query\": q})\nif r.status_code == 500:\n    logging.error(\"crawl failed: %s\", r.json().get(\"detail\"))  # actionable crawler error\n    # fix URL / egress / browser deps based on the message","handlingStrategy":"validation","validationCode":"from urllib.parse import urlparse\n\ndef url_likely_crawlable(url: str) -> bool:\n    p = urlparse(url if \"//\" in url else \"https://\" + url)\n    return p.scheme in (\"http\", \"https\") and bool(p.netloc) and \".\" in p.netloc","typeGuard":null,"tryCatchPattern":"r = await client.post(\"/q\", json=body)\nif r.status_code == 500:\n    detail = r.json().get(\"detail\", \"\")\n    if \"ERR_NAME_NOT_RESOLVED\" in detail or \"Timeout\" in detail:\n        await asyncio.sleep(2)  # transient network — retry once\n        r = await client.post(\"/q\", json=body)\n    else:\n        raise RuntimeError(f\"crawl failed: {detail}\")","preventionTips":["Verify URLs resolve before submitting (DNS check or HEAD request)","Include the scheme explicitly in submitted URLs to avoid mis-normalization","Treat the 500 detail as the crawler's diagnosis, not server failure"],"tags":["http-500","crawl-failure","error-message","server"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}