unclecode/crawl4ai · critical · Exception

CDP endpoint at {cdp_url} is not ready after startup

Error message

CDP endpoint at {cdp_url} is not ready after startup

What it means

When connecting over CDP (config.cdp_url set, or use_managed_browser=True), BrowserManager probes the endpoint with _verify_cdp_ready(cdp_url) before calling playwright.chromium.connect_over_cdp. If the DevTools endpoint does not answer within its retry window, this Exception aborts connection. Note it fires before any Playwright error, so the real state is: nothing is listening / not ready at that URL.

Source

Thrown at crawl4ai/browser_manager.py:889

            self._launched_persistent = True

            await self.setup_context(self.default_context)

            # Set the browser endpoint key for global page tracking
            self._browser_endpoint_key = self._compute_browser_endpoint_key()
            if self._browser_endpoint_key not in BrowserManager._global_pages_in_use:
                BrowserManager._global_pages_in_use[self._browser_endpoint_key] = set()
            return

        if self.config.cdp_url or self.config.use_managed_browser:
            self.config.use_managed_browser = True

            if not self._using_cached_cdp:
                cdp_url = await self.managed_browser.start() if not self.config.cdp_url else self.config.cdp_url

                # Add CDP endpoint verification before connecting
                if not await self._verify_cdp_ready(cdp_url):
                    raise Exception(f"CDP endpoint at {cdp_url} is not ready after startup")

                self.browser = await self.playwright.chromium.connect_over_cdp(cdp_url)

            contexts = self.browser.contexts

            # If browser_context_id is provided, we're using a pre-created context
            if self.config.browser_context_id:
                if self.logger:
                    self.logger.debug(
                        f"Using pre-existing browser context: {self.config.browser_context_id}",
                        tag="BROWSER"
                    )
                # When connecting to a pre-created context, it should be in contexts
                if contexts:
                    self.default_context = contexts[0]
                    if self.logger:
                        self.logger.debug(
                            f"Found {len(contexts)} existing context(s), using first one",

View on GitHub (pinned to 7e80152142)

Solutions

  1. Start Chrome with debugging enabled and a separate profile: chrome --remote-debugging-port=9222 --user-data-dir=/tmp/chrome-debug, then point cdp_url at http://localhost:9222.
  2. Verify the endpoint yourself: curl http://localhost:9222/json/version — it must return JSON before you start the crawler.
  3. If using the managed browser (no cdp_url), check that the browser binary launched at all (see the 'Failed to start browser' error) and that debugging_port is free.
  4. Fix host/port in cdp_url (scheme http, correct container hostnames in docker-compose).

Example fix

# before
BrowserConfig(cdp_url='http://localhost:9222')  # nothing listening -> not ready

# after
# terminal: google-chrome --remote-debugging-port=9222 --user-data-dir=/tmp/chrome-debug
# curl http://localhost:9222/json/version  # sanity check
BrowserConfig(cdp_url='http://localhost:9222')
Defensive patterns

Strategy: retry

Validate before calling

import urllib.request, json
def cdp_ready(cdp_url: str, timeout: float = 2.0) -> bool:
    try:
        with urllib.request.urlopen(f'{cdp_url.rstrip("/")}/json/version', timeout=timeout) as r:
            return r.status == 200 and json.loads(r.read()).get('Browser')
    except Exception:
        return False

Try / catch

try:
    async with AsyncWebCrawler(config=BrowserConfig(cdp_url=cdp_url)) as c:
        ...
except Exception as e:
    if 'not ready after startup' in str(e):
        # wait for the external Chrome to boot and retry once
        await asyncio.sleep(5)
        async with AsyncWebCrawler(config=BrowserConfig(cdp_url=cdp_url)) as c:
            ...
    else:
        raise

Prevention

When it happens

Trigger: config.cdp_url='http://localhost:9222' but no Chrome started with --remote-debugging-port=9222; the managed browser subprocess crashed right after start; CDP bound to 127.0.0.1 while you connect to another host/port; firewall or wrong port; Chrome started with the new --remote-debugging-pipe instead of port.

Common situations: Connecting to an external/pre-launched Chrome that was started without the debugging flag; race where the managed browser is still booting; Docker port-mapping mismatches; Chrome 111+ requiring --remote-debugging-port explicitly plus a non-default user-data-dir.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/e7f59960740aeef5. Report an issue: GitHub.