{"record":{"id":"5505c171b9518595","repo":"NanmiCoder/MediaCrawler","slug":"access-denied","errorCode":null,"errorMessage":"Access denied","messagePattern":"Access denied","errorType":"http","errorClass":"HTTPException","httpStatus":403,"severity":"error","filePath":"api/routers/data.py","lineNumber":113,"sourceCode":"    return {\"files\": files}\n\n\n@router.get(\"/files/{file_path:path}\")\nasync def get_file_content(file_path: str, preview: bool = True, limit: int = 100):\n    \"\"\"Get file content or preview\"\"\"\n    full_path = DATA_DIR / file_path\n\n    if not full_path.exists():\n        raise HTTPException(status_code=404, detail=\"File not found\")\n\n    if not full_path.is_file():\n        raise HTTPException(status_code=400, detail=\"Not a file\")\n\n    # Security check: ensure within DATA_DIR\n    try:\n        full_path.resolve().relative_to(DATA_DIR.resolve())\n    except ValueError:\n        raise HTTPException(status_code=403, detail=\"Access denied\")\n\n    if preview:\n        # Return preview data\n        try:\n            if full_path.suffix == \".json\":\n                with open(full_path, \"r\", encoding=\"utf-8\") as f:\n                    data = json.load(f)\n                    if isinstance(data, list):\n                        return {\"data\": data[:limit], \"total\": len(data)}\n                    return {\"data\": data, \"total\": 1}\n            elif full_path.suffix == \".csv\":\n                import csv\n                with open(full_path, \"r\", encoding=\"utf-8\") as f:\n                    reader = csv.DictReader(f)\n                    rows = []\n                    for i, row in enumerate(reader):\n                        if i >= limit:\n                            break","sourceCodeStart":95,"sourceCodeEnd":131,"githubUrl":"https://github.com/NanmiCoder/MediaCrawler/blob/d6f7c5bb906b6dac40ddf343ef9e26438a3de092/api/routers/data.py#L95-L131","documentation":"Raised in crawler_util.find_login_qrcode when the login page's QR <img> src points at an http(s) URL but fetching it returned a non-200 status. The scraper needs the image bytes to base64-encode for later decoding, so any non-200 aborts with the response body attached. The enclosing except then prints the error and returns an empty string, so upstream this manifests as an empty QR string rather than an exception.","triggerScenarios":"GET of the QR image URL returns 403/404/302-to-login: the platform blocks requests whose headers/cookies differ from the browser session; the URL is signed and expired by the time it is fetched; the page <img> src attribute literally contains a non-http placeholder that still matches 'http' via data URI edge cases; missing Referer/Cookie headers.","commonSituations":"Anti-bot CDNs (e.g. WAF rules on qrcode endpoints) rejecting bare httpx requests; using a proxied httpx client whose exit IP is geo-blocked; the login session cookie captured earlier expired; platform changed QR endpoint to require auth headers.","solutions":["Replay the request with the browser's own headers and cookies: pass the page's cookies and Referer (page.url) to client.get","Fetch the image inside the page instead of via httpx: `await elements.screenshot()` or read the loaded image via CDP — bypasses URL-level anti-bot entirely","Retry with backoff; a single 403 from a CDN is often transient rate limiting","Log resp.status_code alongside resp.text — a 403 vs 404 vs 302 points to header vs URL-expiry causes respectively"],"exampleFix":"// before\nresp = await client.get(login_qrcode_img, headers={\"User-Agent\": get_user_agent()})\nif resp.status_code == 200:\n    ...\nraise Exception(f\"fetch login image url failed, response message:{resp.text}\")\n\n// after\ncookies = {c[\"name\"]: c[\"value\"] for c in await page.context.cookies()}\nresp = await client.get(\n    login_qrcode_img,\n    headers={\n        \"User-Agent\": get_user_agent(),\n        \"Referer\": page.url,\n    },\n    cookies=cookies,\n)\nif resp.status_code != 200:\n    # fall back to screenshotting the element the browser already loaded\n    return base64.b64encode(await elements.screenshot()).decode(\"utf-8\")","handlingStrategy":"fallback","validationCode":"async def fetch_qrcode_bytes(page, img_el) -> bytes | None:\n    src = str(await img_el.get_property(\"src\"))\n    if not src.startswith((\"http://\", \"https://\")):\n        return None  # data URI — use as-is, no HTTP fetch needed\n    cookies = {c[\"name\"]: c[\"value\"] for c in await page.context.cookies()}\n    async with make_async_client(follow_redirects=True) as client:\n        resp = await client.get(src, headers={\"User-Agent\": get_user_agent(), \"Referer\": page.url}, cookies=cookies)\n    return resp.content if resp.status_code == 200 else None","typeGuard":"def is_http_img_src(src: str) -> bool:\n    return isinstance(src, str) and (src.startswith(\"http://\") or src.startswith(\"https://\"))","tryCatchPattern":"try:\n    b64 = await find_login_qrcode(page, selector)\nexcept Exception:\n    b64 = \"\"\nif not b64:\n    # HTTP fetch failed — the browser already rendered the QR; screenshot it instead\n    el = await page.wait_for_selector(selector)\n    b64 = base64.b64encode(await el.screenshot()).decode(\"utf-8\")","preventionTips":["Prefer screenshotting the <img> element over re-fetching its URL — the browser already defeated the anti-bot checks","Always send the page's cookies and Referer when re-fetching media discovered on a logged-in page","Check the returned QR string is non-empty before proceeding; find_login_qrcode silently returns '' on failure"],"tags":["qrcode","login","httpx","anti-bot","crawler","image"],"backgroundTag":null,"analyzedSha":"d6f7c5bb906b6dac40ddf343ef9e26438a3de092","analyzedAt":"2026-08-15T01:39:07.505Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}