NanmiCoder/MediaCrawler · error · HTTPException

Crawler is already running

Error message

Crawler is already running

What it means

Raised by CDPBrowserManager._get_browser_websocket_url after the Chrome DevTools Protocol /json/version endpoint answered HTTP 200 but the JSON body contained no 'webSocketDebuggerUrl' key. The manager needs that WebSocket endpoint to hand to playwright.chromium.connect_over_cdp. Chrome 136+ intentionally omits/hides the DevTools HTTP endpoints for browsers launched with a persistent user-data-dir, so a 200 response without the WS URL is now a common outcome.

Source

Thrown at api/routers/crawler.py:34

# 详细许可条款请参阅项目根目录下的LICENSE文件。
# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。

from fastapi import APIRouter, HTTPException

from ..schemas import CrawlerStartRequest, CrawlerStatusResponse
from ..services import crawler_manager

router = APIRouter(prefix="/crawler", tags=["crawler"])


@router.post("/start")
async def start_crawler(request: CrawlerStartRequest):
    """Start crawler task"""
    success = await crawler_manager.start(request)
    if not success:
        # Handle concurrent/duplicate requests: if process is already running, return 400 instead of 500
        if crawler_manager.process and crawler_manager.process.poll() is None:
            raise HTTPException(status_code=400, detail="Crawler is already running")
        raise HTTPException(status_code=500, detail="Failed to start crawler")

    return {"status": "ok", "message": "Crawler started successfully"}


@router.post("/stop")
async def stop_crawler():
    """Stop crawler task"""
    success = await crawler_manager.stop()
    if not success:
        # Handle concurrent/duplicate requests: if process already exited/doesn't exist, return 400 instead of 500
        if not crawler_manager.process or crawler_manager.process.poll() is not None:
            raise HTTPException(status_code=400, detail="No crawler is running")
        raise HTTPException(status_code=500, detail="Failed to stop crawler")

    return {"status": "ok", "message": "Crawler stopped successfully"}

View on GitHub (pinned to d6f7c5bb90)

Solutions

1. If targeting an existing browser, enable the config.CDP_CONNECT_EXISTING path (see _connect_via_cdp) which skips /json/version and calls connect_over_cdp with the http endpoint directly
2. Launch the debug Chrome with a dedicated empty user-data-dir (e.g. --user-data-dir=/tmp/chrome-debug) so /json/version exposes webSocketDebuggerUrl again
3. Retry the fetch for a few seconds after browser start — the endpoint can briefly return a stub body while DevTools initializes
4. If you must use the daily profile on Chrome 136+, pass the CDP endpoint URL (http://localhost:{port}) straight to playwright.chromium.connect_over_cdp instead of parsing /json/version

Example fix

// before
ws_url = data.get("webSocketDebuggerUrl")
if ws_url:
    return ws_url
else:
    raise RuntimeError("webSocketDebuggerUrl not found")

// after
ws_url = data.get("webSocketDebuggerUrl")
if ws_url:
    return ws_url
# Chrome 136+ on existing profiles: fall back to the raw HTTP endpoint,
# which connect_over_cdp accepts directly
return f"http://localhost:{debug_port}"

async def get_ws_url_or_endpoint(debug_port: int) -> str:
    async with make_async_client(follow_redirects=True) as client:
        resp = await client.get(f"http://localhost:{debug_port}/json/version", timeout=10)
    if resp.status_code == 200:
        ws = resp.json().get("webSocketDebuggerUrl")
        if ws:
            return ws
    # Chrome 136+ / existing profile: connect_over_cdp accepts the HTTP endpoint
    return f"http://localhost:{debug_port}"

def has_ws_url(data: dict) -> bool:
    return isinstance(data, dict) and isinstance(data.get("webSocketDebuggerUrl"), str) and data["webSocketDebuggerUrl"].startswith(("ws://", "wss://"))

try:
    ws_url = await manager._get_browser_websocket_url(port)
except RuntimeError:
    ws_url = f"http://localhost:{port}"  # Chrome 136+ fallback
# caller must still validate connection afterwards:
assert manager.browser and manager.browser.is_connected()

When it happens

Trigger: GET http://localhost:{debug_port}/json/version returns 200 with JSON lacking 'webSocketDebuggerUrl' — typically when connecting to an existing Chrome 136+ instance started with --remote-debugging-port on a non-empty user profile, or when another client already holds the DevTools socket.

Common situations: Upgrading Chrome past v136 while keeping the 'attach to my daily browser' workflow; launching Chrome with --user-data-dir pointing at the default profile; anti-debugging enterprise Chrome builds; connecting to a browser whose DevTools server was started with --remote-allow-origins restrictions.

Related errors

• File not found
• No crawler is running
• Not a file
• Cannot connect to existing browser on port {self.debug_port}
• Cannot find available port, tried {start_port} to {port-1}

AI-assisted analysis of NanmiCoder/MediaCrawler@d6f7c5bb90 (2026-08-15). Data as JSON: /api/errors/e8059ab60d4298db. Report an issue: GitHub.