{"record":{"id":"49d92c24e7e374f5","repo":"NanmiCoder/MediaCrawler","slug":"unsupported-file-type-for-preview","errorCode":null,"errorMessage":"Unsupported file type for preview","messagePattern":"Unsupported file type for preview","errorType":"http","errorClass":"HTTPException","httpStatus":400,"severity":"error","filePath":"api/routers/data.py","lineNumber":152,"sourceCode":"                    f.seek(0)\n                    total = sum(1 for _ in f) - 1\n                    return {\"data\": rows, \"total\": total}\n            elif full_path.suffix.lower() in (\".xlsx\", \".xls\"):\n                import pandas as pd\n                # Read first limit rows\n                df = pd.read_excel(full_path, nrows=limit)\n                # Get total row count (only read first column to save memory)\n                df_count = pd.read_excel(full_path, usecols=[0])\n                total = len(df_count)\n                # Convert to list of dictionaries, handle NaN values\n                rows = df.where(pd.notnull(df), None).to_dict(orient='records')\n                return {\n                    \"data\": rows,\n                    \"total\": total,\n                    \"columns\": list(df.columns)\n                }\n            else:\n                raise HTTPException(status_code=400, detail=\"Unsupported file type for preview\")\n        except json.JSONDecodeError:\n            raise HTTPException(status_code=400, detail=\"Invalid JSON file\")\n        except Exception as e:\n            raise HTTPException(status_code=500, detail=str(e))\n    else:\n        # Return file download\n        return FileResponse(\n            path=full_path,\n            filename=full_path.name,\n            media_type=\"application/octet-stream\"\n        )\n\n\n@router.get(\"/download/{file_path:path}\")\nasync def download_file(file_path: str):\n    \"\"\"Download file\"\"\"\n    full_path = DATA_DIR / file_path\n","sourceCodeStart":134,"sourceCodeEnd":170,"githubUrl":"https://github.com/NanmiCoder/MediaCrawler/blob/d6f7c5bb906b6dac40ddf343ef9e26438a3de092/api/routers/data.py#L134-L170","documentation":"Raised in slider_util when downloading a slider-captcha background/puzzle image via plain httpx.get and the server answers non-200. The downloaded bytes are needed for OpenCV gap detection (cv2.imdecode → resize → imwrite), so a failed download aborts before any CV processing. Note ./temp_image/ must already exist or cv2.imwrite will silently fail later even on success paths.","triggerScenarios":"httpx.get(img, headers=headers) with a hard-coded Chrome/91 User-Agent returns 403 (WAF fingerprints the stale UA and plain httpx TLS fingerprint) or 404/410 (signed, expired captcha URL); missing Referer/cookies the captcha endpoint requires.","commonSituations":"Captcha CDN rejecting non-browser TLS fingerprints (httpx vs real Chrome JA3); expired signed URLs because the code slept between page load and image fetch; platform rotated captcha endpoints; running through a datacenter-IP proxy that the captcha provider blocks.","solutions":["Update the request headers: use a current Chrome User-Agent plus Referer set to the challenge page URL, and attach the browser session cookies","Better: extract the image through the already-authenticated Playwright page (page.request.get or fetch via page.evaluate) so TLS fingerprint, cookies, and headers match the real browser","Retry the download a few times — captcha CDNs often 403 rate-limit single bursts","If captcha URL came from an earlier page load, re-scrape it immediately before download to avoid signed-URL expiry"],"exampleFix":"// before\nimg_res = httpx.get(img, headers=headers)\nif img_res.status_code == 200:\n    ...\nelse:\n    raise Exception(f\"Failed to save {img_type} image\")\n\n// after\nresp = await page.request.get(img, headers={\"Referer\": page.url})\nif resp.status != 200:\n    raise Exception(f\"Failed to save {img_type} image: HTTP {resp.status}\")\nimage = cv2.imdecode(\n    np.asarray(bytearray(await resp.body()), dtype=\"uint8\"),\n    cv2.IMREAD_COLOR,\n)","handlingStrategy":"retry","validationCode":"def fetch_slider_image(img: str, referer: str, attempts: int = 3) -> bytes | None:\n    headers = {\n        \"User-Agent\": get_user_agent(),  # current UA, not hard-coded Chrome/91\n        \"Referer\": referer,\n    }\n    for _ in range(attempts):\n        res = httpx.get(img, headers=headers, follow_redirects=True)\n        if res.status_code == 200:\n            return res.content\n    return None  # caller falls back to page.request.get / element screenshot","typeGuard":null,"tryCatchPattern":"try:\n    img_path = save_slider_img(img_url, img_type, resize=resize)\nexcept Exception as e:\n    if \"Failed to save\" in str(e):\n        # fall back to in-browser fetch: same TLS fingerprint and cookies as the challenge page\n        img_path = save_via_page_request(page, img_url, img_type, resize)\n    else:\n        raise","preventionTips":["Use the Playwright page's request context (page.request.get) for captcha assets so TLS fingerprint, cookies, and headers match the real browser","Keep the User-Agent header in sync with the browser actually driving the page — stale hard-coded UAs get 403'd","Create ./temp_image/ at startup (os.makedirs(exist_ok=True)) — cv2.imwrite fails silently otherwise","Include resp.status_code in the exception message so 403 vs 404 causes are distinguishable in logs"],"tags":["slider-captcha","opencv","httpx","anti-bot","image","crawler"],"backgroundTag":null,"analyzedSha":"d6f7c5bb906b6dac40ddf343ef9e26438a3de092","analyzedAt":"2026-08-15T01:39:07.505Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}