unclecode/crawl4ai · error · RequestError

Server error {e.response.status_code}: {error_msg}

Error message

Server error {e.response.status_code}: {error_msg}

What it means

Raised by Crawl4aiDockerClient._request when the server returns a non-2xx status (raise_for_status triggers httpx.HTTPStatusError). The client extracts the 'detail' field from a JSON error body if present, otherwise uses the raw status line, and raises a RequestError('Server error {status}: {detail}'). This is the server explicitly rejecting the request.

Source

Thrown at crawl4ai/docker_client.py:124

        return request_data

    async def _request(self, method: str, endpoint: str, **kwargs) -> httpx.Response:
        """Make an HTTP request with error handling."""
        url = urljoin(self.base_url, endpoint)
        try:
            response = await self._http_client.request(method, url, **kwargs)
            response.raise_for_status()
            return response
        except httpx.TimeoutException as e:
            raise ConnectionError(f"Request timed out: {str(e)}")
        except httpx.RequestError as e:
            raise ConnectionError(f"Failed to connect: {str(e)}")
        except httpx.HTTPStatusError as e:
            error_msg = (e.response.json().get("detail", str(e)) 
                        if "application/json" in e.response.headers.get("content-type", "") 
                        else str(e))
            raise RequestError(f"Server error {e.response.status_code}: {error_msg}")

    async def crawl(
        self,
        urls: List[str],
        browser_config: Optional[BrowserConfig] = None,
        crawler_config: Optional[CrawlerRunConfig] = None,
        hooks: Optional[Union[Dict[str, Callable], Dict[str, str]]] = None,
        hooks_timeout: int = 30
    ) -> Union[CrawlResult, List[CrawlResult], AsyncGenerator[CrawlResult, None]]:
        """
        Execute a crawl operation.

        Args:
            urls: List of URLs to crawl
            browser_config: Browser configuration
            crawler_config: Crawler configuration
            hooks: Optional hooks - can be either:
                   - Dict[str, Callable]: Function objects that will be converted to strings

View on GitHub (pinned to 7e80152142)

Solutions

1. Read the detail text — FastAPI 422 detail names the exact invalid config field
2. Call get_schema() (when available) to see which config keys the server version accepts
3. Ensure authenticate() was called before making requests
4. Pin the Docker server image to the same version as the installed crawl4ai client

Example fix

// before
results = await client.crawl(urls, browser_config=BrowserConfig(some_new_field=1))
# Server error 422: {'detail': [...]}

// after
schema = await client.get_schema()  # inspect accepted fields
results = await client.crawl(urls, browser_config=BrowserConfig(headless=True))

schema = await client.get_schema()  # accepted fields per config type
# intersect your config kwargs with schema before sending
allowed = schema["GET"]["/crawl"]["parameters"]["browser_config"]
bc_kwargs = {k: v for k, v in my_kwargs.items() if k in allowed}

try:
    results = await client.crawl(urls, browser_config=b, crawler_config=c)
except RequestError as e:
    status = str(e).split(':')[0].replace('Server error ', '')
    if status == '401':
        await client.authenticate(email)  # re-auth then retry once
    elif status == '422':
        log.error("config rejected: %s", e)  # fix fields named in detail
    else:
        raise

When it happens

Trigger: POSTing to /crawl with browser_config/crawler_config payloads the server cannot validate (e.g. 422 with detail listing invalid fields); calling an endpoint without prior authentication resulting in 401; 500 from a server-side crawl crash; version mismatch between client and server API.

Common situations: Passing configs with fields unknown to the server's schema (client newer/older than the Docker image); forgetting to call authenticate() so the Authorization header is missing; the server running an older image lacking newer endpoints like /schema.

Related errors

• Container not found: ${config.container_selector}
• Timeout after {timeout}ms waiting for selector '{wait_for}'
• [NSTProxy] token and channel_id are required
• [NSTProxy] Invalid protocol: {protocol}
• concurrency must be positive

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/f2a1027817ae1ea3. Report an issue: GitHub.