NanmiCoder/MediaCrawler · error · HTTPException

Unsupported file type for preview

Error message

Unsupported file type for preview

What it means

Raised in slider_util when downloading a slider-captcha background/puzzle image via plain httpx.get and the server answers non-200. The downloaded bytes are needed for OpenCV gap detection (cv2.imdecode → resize → imwrite), so a failed download aborts before any CV processing. Note ./temp_image/ must already exist or cv2.imwrite will silently fail later even on success paths.

Source

Thrown at api/routers/data.py:152

                    f.seek(0)
                    total = sum(1 for _ in f) - 1
                    return {"data": rows, "total": total}
            elif full_path.suffix.lower() in (".xlsx", ".xls"):
                import pandas as pd
                # Read first limit rows
                df = pd.read_excel(full_path, nrows=limit)
                # Get total row count (only read first column to save memory)
                df_count = pd.read_excel(full_path, usecols=[0])
                total = len(df_count)
                # Convert to list of dictionaries, handle NaN values
                rows = df.where(pd.notnull(df), None).to_dict(orient='records')
                return {
                    "data": rows,
                    "total": total,
                    "columns": list(df.columns)
                }
            else:
                raise HTTPException(status_code=400, detail="Unsupported file type for preview")
        except json.JSONDecodeError:
            raise HTTPException(status_code=400, detail="Invalid JSON file")
        except Exception as e:
            raise HTTPException(status_code=500, detail=str(e))
    else:
        # Return file download
        return FileResponse(
            path=full_path,
            filename=full_path.name,
            media_type="application/octet-stream"
        )


@router.get("/download/{file_path:path}")
async def download_file(file_path: str):
    """Download file"""
    full_path = DATA_DIR / file_path

View on GitHub (pinned to d6f7c5bb90)

Solutions

1. Update the request headers: use a current Chrome User-Agent plus Referer set to the challenge page URL, and attach the browser session cookies
2. Better: extract the image through the already-authenticated Playwright page (page.request.get or fetch via page.evaluate) so TLS fingerprint, cookies, and headers match the real browser
3. Retry the download a few times — captcha CDNs often 403 rate-limit single bursts
4. If captcha URL came from an earlier page load, re-scrape it immediately before download to avoid signed-URL expiry

Example fix

// before
img_res = httpx.get(img, headers=headers)
if img_res.status_code == 200:
    ...
else:
    raise Exception(f"Failed to save {img_type} image")

// after
resp = await page.request.get(img, headers={"Referer": page.url})
if resp.status != 200:
    raise Exception(f"Failed to save {img_type} image: HTTP {resp.status}")
image = cv2.imdecode(
    np.asarray(bytearray(await resp.body()), dtype="uint8"),
    cv2.IMREAD_COLOR,
)

def fetch_slider_image(img: str, referer: str, attempts: int = 3) -> bytes | None:
    headers = {
        "User-Agent": get_user_agent(),  # current UA, not hard-coded Chrome/91
        "Referer": referer,
    }
    for _ in range(attempts):
        res = httpx.get(img, headers=headers, follow_redirects=True)
        if res.status_code == 200:
            return res.content
    return None  # caller falls back to page.request.get / element screenshot

try:
    img_path = save_slider_img(img_url, img_type, resize=resize)
except Exception as e:
    if "Failed to save" in str(e):
        # fall back to in-browser fetch: same TLS fingerprint and cookies as the challenge page
        img_path = save_via_page_request(page, img_url, img_type, resize)
    else:
        raise

When it happens

Trigger: httpx.get(img, headers=headers) with a hard-coded Chrome/91 User-Agent returns 403 (WAF fingerprints the stale UA and plain httpx TLS fingerprint) or 404/410 (signed, expired captcha URL); missing Referer/cookies the captcha endpoint requires.

Common situations: Captcha CDN rejecting non-browser TLS fingerprints (httpx vs real Chrome JA3); expired signed URLs because the code slept between page load and image fetch; platform rotated captcha endpoints; running through a datacenter-IP proxy that the captcha provider blocks.

Related errors

• Access denied
• get weibo detail err: {response.text}
• XHS request blocked with HTTP {response.status_code}
• CAPTCHA appeared, request failed, Verifytype: {verify_type},
• 300012

AI-assisted analysis of NanmiCoder/MediaCrawler@d6f7c5bb90 (2026-08-15). Data as JSON: /api/errors/49d92c24e7e374f5. Report an issue: GitHub.