NanmiCoder/MediaCrawler · error · HTTPException

No crawler is running

Error message

No crawler is running

What it means

Raised when the DevTools /json/version HTTP request completes but returns a non-200 status. It means something answered on the debug port, yet it is not a healthy CDP endpoint — the error text embeds the status code and body (note: the message is an un-expanded f-string in the raise site's metadata, but at runtime it carries the real status and response text).

Source

Thrown at api/routers/crawler.py:47

    """Start crawler task"""
    success = await crawler_manager.start(request)
    if not success:
        # Handle concurrent/duplicate requests: if process is already running, return 400 instead of 500
        if crawler_manager.process and crawler_manager.process.poll() is None:
            raise HTTPException(status_code=400, detail="Crawler is already running")
        raise HTTPException(status_code=500, detail="Failed to start crawler")

    return {"status": "ok", "message": "Crawler started successfully"}


@router.post("/stop")
async def stop_crawler():
    """Stop crawler task"""
    success = await crawler_manager.stop()
    if not success:
        # Handle concurrent/duplicate requests: if process already exited/doesn't exist, return 400 instead of 500
        if not crawler_manager.process or crawler_manager.process.poll() is not None:
            raise HTTPException(status_code=400, detail="No crawler is running")
        raise HTTPException(status_code=500, detail="Failed to stop crawler")

    return {"status": "ok", "message": "Crawler stopped successfully"}


@router.get("/status", response_model=CrawlerStatusResponse)
async def get_crawler_status():
    """Get crawler status"""
    return crawler_manager.get_status()


@router.get("/logs")
async def get_logs(limit: int = 100):
    """Get recent logs"""
    logs = crawler_manager.logs[-limit:] if limit > 0 else crawler_manager.logs
    return {"logs": [log.model_dump() for log in logs]}

View on GitHub (pinned to d6f7c5bb90)

Solutions

1. Verify the endpoint manually: curl http://localhost:{debug_port}/json/version — a healthy browser returns JSON with Browser/webSocketDebuggerUrl fields
2. Confirm Chrome was launched with --remote-debugging-port={debug_port} and check the process is alive (ps aux | grep chrome)
3. Clear proxy env vars (HTTP_PROXY/HTTPS_PROXY/NO_PROXY) or build the async client with trust_env=False so localhost is never proxied
4. Retry with backoff for a few seconds after browser launch to ride out DevTools startup latency
5. Pick a different debug_port if another process owns it

Example fix

// before
response = await client.get(
    f"http://localhost:{debug_port}/json/version", timeout=10
)
if response.status_code == 200:
    ...
else:
    raise RuntimeError(f"HTTP {response.status_code}: {response.text}")

// after
import asyncio
for _ in range(10):
    response = await client.get(
        f"http://localhost:{debug_port}/json/version",
        timeout=10,
        # never let a system proxy touch localhost
        headers={"Host": "localhost"},
    )
    if response.status_code == 200:
        return response.json().get("webSocketDebuggerUrl")
    await asyncio.sleep(1)
raise RuntimeError(f"HTTP {response.status_code}: {response.text}")

async def wait_for_devtools(debug_port: int, attempts: int = 10, delay: float = 1.0) -> dict:
    async with make_async_client(follow_redirects=True) as client:
        for _ in range(attempts):
            try:
                resp = await client.get(f"http://localhost:{debug_port}/json/version", timeout=5)
                if resp.status_code == 200:
                    return resp.json()
            except httpx.HTTPError:
                pass
            await asyncio.sleep(delay)
    raise RuntimeError(f"DevTools not healthy on port {debug_port}")

try:
    version = await wait_for_devtools(port)
except RuntimeError as e:
    logger.error(f"DevTools endpoint unhealthy: {e}")
    raise  # do not proceed with a half-initialized browser

When it happens

Trigger: GET http://localhost:{debug_port}/json/version returning 404/500/503 — e.g. the port is occupied by a non-DevTools service, a proxy answered, or Chrome is still starting up and the DevTools handler is not yet mounted.

Common situations: debug_port typo or collision with another service; a system HTTP proxy intercepting localhost requests (httpx honors proxies env vars); querying the endpoint immediately after spawning Chrome before DevTools is listening; Chrome crashed at startup (bad flags) so an error page or nothing responds.

Related errors

• Crawler is already running
• get ip error from proxy provider and status code not 200 ...
• Cannot connect to existing browser on port {self.debug_port}
• File not found
• Not a file

AI-assisted analysis of NanmiCoder/MediaCrawler@d6f7c5bb90 (2026-08-15). Data as JSON: /api/errors/00541b81fb10e7af. Report an issue: GitHub.