{"record":{"id":"2093780fb80d4812","repo":"unclecode/crawl4ai","slug":"crawl-failed-result-data-get-msg-unknown-err","errorCode":null,"errorMessage":"Crawl failed: {result_data.get('msg', 'Unknown error')}","messagePattern":"Crawl failed: (.+?)","errorType":"exception","errorClass":"RequestError","httpStatus":null,"severity":"error","filePath":"crawl4ai/docker_client.py","lineNumber":186,"sourceCode":"                async with self._http_client.stream(\"POST\", f\"{self.base_url}/crawl/stream\", json=data) as response:\n                    response.raise_for_status()\n                    async for line in response.aiter_lines():\n                        if line.strip():\n                            result = json.loads(line)\n                            if \"error\" in result:\n                                self.logger.error_status(url=result.get(\"url\", \"unknown\"), error=result[\"error\"])\n                                continue\n                            self.logger.url_status(url=result.get(\"url\", \"unknown\"), success=True, timing=result.get(\"timing\", 0.0))\n                            if result.get(\"status\") == \"completed\":\n                                continue\n                            else:\n                                yield CrawlResult(**result)\n            return stream_results()\n\n        response = await self._request(\"POST\", \"/crawl\", json=data, timeout=hooks_timeout)\n        result_data = response.json()\n        if not result_data.get(\"success\", False):\n            raise RequestError(f\"Crawl failed: {result_data.get('msg', 'Unknown error')}\")\n\n        results = [CrawlResult(**r) for r in result_data.get(\"results\", [])]\n        self.logger.success(f\"Crawl completed with {len(results)} results\", tag=\"CRAWL\")\n        return results[0] if len(results) == 1 else results\n\n    async def get_schema(self) -> Dict[str, Any]:\n        \"\"\"Retrieve configuration schemas.\"\"\"\n        response = await self._request(\"GET\", \"/schema\")\n        return response.json()\n\n    async def close(self) -> None:\n        \"\"\"Close the HTTP client session.\"\"\"\n        self.logger.info(\"Closing client\", tag=\"CLOSE\")\n        await self._http_client.aclose()\n\n    async def __aenter__(self) -> \"Crawl4aiDockerClient\":\n        return self\n","sourceCodeStart":168,"sourceCodeEnd":204,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/docker_client.py#L168-L204","documentation":"Raised after a successful POST /crawl when the server responds 2xx but with JSON body success=false. The msg field from the response (or 'Unknown error' if absent) is surfaced in RequestError('Crawl failed: {msg}'). This is the server reporting that the crawl job itself failed, as opposed to a transport error.","triggerScenarios":"The server-side crawl of one or more URLs fails: invalid URLs, browser launch failure inside the container, or a rejected payload that passes HTTP validation but fails job execution. The response JSON contains success=false and a msg describing the server-side failure.","commonSituations":"Passing malformed URLs; the container lacks browser dependencies or resources (shared memory, sandbox flags); server bug or unhandled exception during the crawl; sending configs (e.g. magic/simulation modes) the server runtime cannot execute.","solutions":["Print the full msg — it is the server's own failure reason and the fastest diagnosis path","Validate URLs (scheme + host) before sending the batch","Check docker logs for the server-side stack trace at the same timestamp","Reproduce the crawl locally with AsyncWebCrawler to separate server-environment issues from config issues","Update the server image if the msg indicates an unimplemented feature"],"exampleFix":"// before\nresults = await client.crawl([\"not-a-url\"], browser_config=b, crawler_config=c)\n# RequestError: Crawl failed: invalid url\n\n// after\nfrom urllib.parse import urlparse\nurls = [u for u in urls if urlparse(u).scheme in (\"http\", \"https\")]\nresults = await client.crawl(urls, browser_config=b, crawler_config=c)","handlingStrategy":"validation","validationCode":"from urllib.parse import urlparse\n\nurls = [u for u in urls if isinstance(u, str) and urlparse(u).scheme in (\"http\", \"https\") and urlparse(u).netloc]\nassert urls, \"no valid URLs to crawl\"","typeGuard":"def is_crawlable_url(u) -> bool:\n    p = urlparse(u)\n    return p.scheme in (\"http\", \"https\") and bool(p.netloc)","tryCatchPattern":"try:\n    results = await client.crawl(urls, browser_config=b, crawler_config=c)\nexcept RequestError as e:\n    if \"Crawl failed\" in str(e):\n        log.error(\"server reported: %s\", e)  # msg holds the real cause\n    raise","preventionTips":["Print the msg field — it is the server's failure reason","Reproduce failing crawls locally with AsyncWebCrawler","Check docker logs at the failure timestamp for a stack trace"],"tags":["docker-client","crawl-failure","server-error"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}