What causes "File not found" in NanmiCoder/MediaCrawler?

Re-fetch the WS URL immediately before connecting (it is already done here) and retry connect_over_cdp once — stale URLs are the most common cause

NanmiCoder/MediaCrawler · critical · HTTPException

File not found

Error message

File not found

What it means

Raised in _connect_via_cdp when the Playwright CDP attach step did not yield a live connection: either playwright.chromium.connect_over_cdp(ws_url) threw (the outer except logs and re-raises the underlying error), or it returned a browser object whose is_connected() is False. The message is a fallback for the silent-failure branch; most real failures surface as Playwright errors wrapped by the same except block.

Source

Thrown at api/routers/data.py:104

            try:
                files.append(get_file_info(file_path))
            except Exception:
                continue

    # Sort by modification time (newest first)
    files.sort(key=lambda x: x["modified_at"], reverse=True)

    return {"files": files}


@router.get("/files/{file_path:path}")
async def get_file_content(file_path: str, preview: bool = True, limit: int = 100):
    """Get file content or preview"""
    full_path = DATA_DIR / file_path

    if not full_path.exists():
        raise HTTPException(status_code=404, detail="File not found")

    if not full_path.is_file():
        raise HTTPException(status_code=400, detail="Not a file")

    # Security check: ensure within DATA_DIR
    try:
        full_path.resolve().relative_to(DATA_DIR.resolve())
    except ValueError:
        raise HTTPException(status_code=403, detail="Access denied")

    if preview:
        # Return preview data
        try:
            if full_path.suffix == ".json":
                with open(full_path, "r", encoding="utf-8") as f:
                    data = json.load(f)
                    if isinstance(data, list):
                        return {"data": data[:limit], "total": len(data)}

View on GitHub (pinned to d6f7c5bb90)

Solutions

Re-fetch the WS URL immediately before connecting (it is already done here) and retry connect_over_cdp once — stale URLs are the most common cause
Ensure Chrome is launched with --remote-debugging-port and, for WebSocket clients sending Origin headers, --remote-allow-origins=* (or the specific origin)
Check self.browser is closed/killed by a previous run: fully stop old chrome --remote-debugging-port processes and remove stale SingletonLock files in the user-data-dir
Upgrade/align playwright (pip install -U playwright && playwright install chromium) so its CDP client matches modern Chrome
Inspect the logged underlying exception above 'CDP connection failed' — the real cause (ECONNREFUSED, 405, handshake timeout) is in that message

Example fix

// before
self.browser = await playwright.chromium.connect_over_cdp(ws_url)
if self.browser.is_connected():
    ...
else:
    raise RuntimeError("CDP connection failed")

// after
import asyncio
for attempt in range(3):
    try:
        self.browser = await playwright.chromium.connect_over_cdp(ws_url)
        if self.browser.is_connected():
            break
    except Exception:
        if attempt == 2:
            raise
    await asyncio.sleep(2)
    ws_url = await self._get_browser_websocket_url(self.debug_port)
else:
    raise RuntimeError("CDP connection failed")

Defensive patterns

Strategy: retry

Validate before calling

async def connect_with_retry(playwright, manager, attempts: int = 3) -> None:
    for attempt in range(attempts):
        ws_url = await manager._get_browser_websocket_url(manager.debug_port)
        try:
            manager.browser = await playwright.chromium.connect_over_cdp(ws_url)
            if manager.browser.is_connected():
                return
        except Exception:
            if attempt == attempts - 1:
                raise
        await asyncio.sleep(2)

Type guard

def is_connected_browser(browser) -> bool:
    return browser is not None and getattr(browser, "is_connected", lambda: False)()

Try / catch

try:
    await connect_with_retry(playwright, manager)
except PlaywrightError as e:
    logger.error(f"CDP attach failed: {e}")
    # full restart: kill browser, relaunch, reconnect — never reuse stale ws_url
    await manager.stop()
    await manager.start()

Prevention

Always re-fetch the WS URL immediately before connect_over_cdp — never reuse one from a previous browser process
Register browser.on('disconnected') to flag the manager dead instead of limping into later RuntimeError('CDP connection failed')
Launch Chrome with --remote-allow-origins when a WebSocket client with an Origin header connects

When it happens

Trigger: connect_over_cdp to a stale/mismatched webSocketDebuggerUrl; the browser exited between fetching the WS URL and connecting; ws_url points at 127.0.0.1 but the browser listens only on ::1 (or vice versa); Playwright's bundled Chromium revision mismatch; DevTools connection rejected because the browser requires --remote-allow-origins for WebSocket clients.

Common situations: User closes Chrome while the crawler is connecting; reusing a cached ws_url from a previous browser session; running inside Docker/WSL where localhost mapping differs; Playwright version older than the installed Chrome's CDP dialect; headless Chrome without --remote-debugging-port actually set.

Related errors

AI-assisted analysis of NanmiCoder/MediaCrawler@d6f7c5bb90 (2026-08-15). Data as JSON: /api/errors/9d7bb0dcc8152d8d. Report an issue: GitHub.