NanmiCoder/MediaCrawler · error · HTTPException

Access denied

Error message

Access denied

What it means

Raised in crawler_util.find_login_qrcode when the login page's QR <img> src points at an http(s) URL but fetching it returned a non-200 status. The scraper needs the image bytes to base64-encode for later decoding, so any non-200 aborts with the response body attached. The enclosing except then prints the error and returns an empty string, so upstream this manifests as an empty QR string rather than an exception.

Source

Thrown at api/routers/data.py:113

    return {"files": files}


@router.get("/files/{file_path:path}")
async def get_file_content(file_path: str, preview: bool = True, limit: int = 100):
    """Get file content or preview"""
    full_path = DATA_DIR / file_path

    if not full_path.exists():
        raise HTTPException(status_code=404, detail="File not found")

    if not full_path.is_file():
        raise HTTPException(status_code=400, detail="Not a file")

    # Security check: ensure within DATA_DIR
    try:
        full_path.resolve().relative_to(DATA_DIR.resolve())
    except ValueError:
        raise HTTPException(status_code=403, detail="Access denied")

    if preview:
        # Return preview data
        try:
            if full_path.suffix == ".json":
                with open(full_path, "r", encoding="utf-8") as f:
                    data = json.load(f)
                    if isinstance(data, list):
                        return {"data": data[:limit], "total": len(data)}
                    return {"data": data, "total": 1}
            elif full_path.suffix == ".csv":
                import csv
                with open(full_path, "r", encoding="utf-8") as f:
                    reader = csv.DictReader(f)
                    rows = []
                    for i, row in enumerate(reader):
                        if i >= limit:
                            break

View on GitHub (pinned to d6f7c5bb90)

Solutions

  1. Replay the request with the browser's own headers and cookies: pass the page's cookies and Referer (page.url) to client.get
  2. Fetch the image inside the page instead of via httpx: `await elements.screenshot()` or read the loaded image via CDP — bypasses URL-level anti-bot entirely
  3. Retry with backoff; a single 403 from a CDN is often transient rate limiting
  4. Log resp.status_code alongside resp.text — a 403 vs 404 vs 302 points to header vs URL-expiry causes respectively

Example fix

// before
resp = await client.get(login_qrcode_img, headers={"User-Agent": get_user_agent()})
if resp.status_code == 200:
    ...
raise Exception(f"fetch login image url failed, response message:{resp.text}")

// after
cookies = {c["name"]: c["value"] for c in await page.context.cookies()}
resp = await client.get(
    login_qrcode_img,
    headers={
        "User-Agent": get_user_agent(),
        "Referer": page.url,
    },
    cookies=cookies,
)
if resp.status_code != 200:
    # fall back to screenshotting the element the browser already loaded
    return base64.b64encode(await elements.screenshot()).decode("utf-8")
Defensive patterns

Strategy: fallback

Validate before calling

async def fetch_qrcode_bytes(page, img_el) -> bytes | None:
    src = str(await img_el.get_property("src"))
    if not src.startswith(("http://", "https://")):
        return None  # data URI — use as-is, no HTTP fetch needed
    cookies = {c["name"]: c["value"] for c in await page.context.cookies()}
    async with make_async_client(follow_redirects=True) as client:
        resp = await client.get(src, headers={"User-Agent": get_user_agent(), "Referer": page.url}, cookies=cookies)
    return resp.content if resp.status_code == 200 else None

Type guard

def is_http_img_src(src: str) -> bool:
    return isinstance(src, str) and (src.startswith("http://") or src.startswith("https://"))

Try / catch

try:
    b64 = await find_login_qrcode(page, selector)
except Exception:
    b64 = ""
if not b64:
    # HTTP fetch failed — the browser already rendered the QR; screenshot it instead
    el = await page.wait_for_selector(selector)
    b64 = base64.b64encode(await el.screenshot()).decode("utf-8")

Prevention

When it happens

Trigger: GET of the QR image URL returns 403/404/302-to-login: the platform blocks requests whose headers/cookies differ from the browser session; the URL is signed and expired by the time it is fetched; the page <img> src attribute literally contains a non-http placeholder that still matches 'http' via data URI edge cases; missing Referer/Cookie headers.

Common situations: Anti-bot CDNs (e.g. WAF rules on qrcode endpoints) rejecting bare httpx requests; using a proxied httpx client whose exit IP is geo-blocked; the login session cookie captured earlier expired; platform changed QR endpoint to require auth headers.

Understand the failure class

Related errors

AI-assisted analysis of NanmiCoder/MediaCrawler@d6f7c5bb90 (2026-08-15). Data as JSON: /api/errors/5505c171b9518595. Report an issue: GitHub.