What causes "Crawl failed: {result_data.get('msg', 'Unknown error')}" in unclecode/crawl4ai?

Print the full msg — it is the server's own failure reason and the fastest diagnosis path

unclecode/crawl4ai · error · RequestError

Crawl failed: {result_data.get('msg', 'Unknown error')}

Error message

Crawl failed: {result_data.get('msg', 'Unknown error')}

What it means

Raised after a successful POST /crawl when the server responds 2xx but with JSON body success=false. The msg field from the response (or 'Unknown error' if absent) is surfaced in RequestError('Crawl failed: {msg}'). This is the server reporting that the crawl job itself failed, as opposed to a transport error.

Source

Thrown at crawl4ai/docker_client.py:186

                async with self._http_client.stream("POST", f"{self.base_url}/crawl/stream", json=data) as response:
                    response.raise_for_status()
                    async for line in response.aiter_lines():
                        if line.strip():
                            result = json.loads(line)
                            if "error" in result:
                                self.logger.error_status(url=result.get("url", "unknown"), error=result["error"])
                                continue
                            self.logger.url_status(url=result.get("url", "unknown"), success=True, timing=result.get("timing", 0.0))
                            if result.get("status") == "completed":
                                continue
                            else:
                                yield CrawlResult(**result)
            return stream_results()

        response = await self._request("POST", "/crawl", json=data, timeout=hooks_timeout)
        result_data = response.json()
        if not result_data.get("success", False):
            raise RequestError(f"Crawl failed: {result_data.get('msg', 'Unknown error')}")

        results = [CrawlResult(**r) for r in result_data.get("results", [])]
        self.logger.success(f"Crawl completed with {len(results)} results", tag="CRAWL")
        return results[0] if len(results) == 1 else results

    async def get_schema(self) -> Dict[str, Any]:
        """Retrieve configuration schemas."""
        response = await self._request("GET", "/schema")
        return response.json()

    async def close(self) -> None:
        """Close the HTTP client session."""
        self.logger.info("Closing client", tag="CLOSE")
        await self._http_client.aclose()

    async def __aenter__(self) -> "Crawl4aiDockerClient":
        return self

View on GitHub (pinned to 7e80152142)

Solutions

Print the full msg — it is the server's own failure reason and the fastest diagnosis path
Validate URLs (scheme + host) before sending the batch
Check docker logs for the server-side stack trace at the same timestamp
Reproduce the crawl locally with AsyncWebCrawler to separate server-environment issues from config issues
Update the server image if the msg indicates an unimplemented feature

Example fix

// before
results = await client.crawl(["not-a-url"], browser_config=b, crawler_config=c)
# RequestError: Crawl failed: invalid url

// after
from urllib.parse import urlparse
urls = [u for u in urls if urlparse(u).scheme in ("http", "https")]
results = await client.crawl(urls, browser_config=b, crawler_config=c)

Defensive patterns

Strategy: validation

Validate before calling

from urllib.parse import urlparse

urls = [u for u in urls if isinstance(u, str) and urlparse(u).scheme in ("http", "https") and urlparse(u).netloc]
assert urls, "no valid URLs to crawl"

Type guard

def is_crawlable_url(u) -> bool:
    p = urlparse(u)
    return p.scheme in ("http", "https") and bool(p.netloc)

Try / catch

try:
    results = await client.crawl(urls, browser_config=b, crawler_config=c)
except RequestError as e:
    if "Crawl failed" in str(e):
        log.error("server reported: %s", e)  # msg holds the real cause
    raise

Prevention

Print the msg field — it is the server's failure reason
Reproduce failing crawls locally with AsyncWebCrawler
Check docker logs at the failure timestamp for a stack trace

When it happens

Trigger: The server-side crawl of one or more URLs fails: invalid URLs, browser launch failure inside the container, or a rejected payload that passes HTTP validation but fails job execution. The response JSON contains success=false and a msg describing the server-side failure.

Common situations: Passing malformed URLs; the container lacks browser dependencies or resources (shared memory, sandbox flags); server bug or unhandled exception during the crawl; sending configs (e.g. magic/simulation modes) the server runtime cannot execute.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/2093780fb80d4812. Report an issue: GitHub.