NanmiCoder/MediaCrawler · error · DataFetchError

get response code error: {response.status_code}

Error message

get response code error: {response.status_code}

What it means

Raised as DataFetchError by WeiboClient.request (media_platform/weibo/client.py:92) when response.json() throws JSONDecodeError — the Weibo h5 API returned a non-JSON body, classically HTTP 432 on the search endpoint (issue #771). Before raising, the handler attempts self-healing: it navigates the Playwright page to the host and refreshes cookies, expecting upstream retries (tenacity on the caller) to succeed with the new session. So hitting this error means even that refresh path was entered and the raise still propagated.

Source

Thrown at media_platform/weibo/client.py:92

        # Check if proxy is expired before each request
        await self._refresh_proxy_if_expired()

        enable_return_response = kwargs.pop("return_response", False)
        async with make_async_client(proxy=self.proxy) as client:
            response = await client.request(method, url, timeout=self.timeout, **kwargs)

        if enable_return_response:
            return response

        try:
            data: Dict = response.json()
        except json.decoder.JSONDecodeError:
            # issue: #771 Search API returns error 432, retry multiple times + update h5 cookies
            utils.logger.error(f"[WeiboClient.request] request {method}:{url} err code: {response.status_code} res:{response.text}")
            await self.playwright_page.goto(self._host)
            await asyncio.sleep(2)
            await self.update_cookies(browser_context=self.playwright_page.context)
            raise DataFetchError(f"get response code error: {response.status_code}")

        ok_code = data.get("ok")
        if ok_code == 0:  # response error
            utils.logger.error(f"[WeiboClient.request] request {method}:{url} err, res:{data}")
            raise DataFetchError(data.get("msg", "response error"))
        elif ok_code != 1:  # unknown error
            utils.logger.error(f"[WeiboClient.request] request {method}:{url} err, res:{data}")
            raise DataFetchError(data.get("msg", "unknown error"))
        else:  # response right
            return data.get("data", {})

    async def get(self, uri: str, params=None, headers=None, **kwargs) -> Union[Response, Dict]:
        final_uri = uri
        if isinstance(params, dict):
            final_uri = (f"{uri}?"
                         f"{urlencode(params)}")

        if headers is None:

View on GitHub (pinned to d6f7c5bb90)

Solutions

  1. Rely on the built-in retry: callers decorated with tenacity retry will re-run with refreshed cookies — ensure the calling method has retry enabled
  2. Slow down search request frequency (CRAWLER_MAX_SLEEP_SEC) to stop triggering 432
  3. Re-login (qrcode/cookie) to get fresh h5 cookies if retries keep failing
  4. Verify playwright_page is alive so the cookie-refresh goto can actually run
Defensive patterns

Strategy: retry

Try / catch

from tenacity import retry, stop_after_attempt, wait_fixed
from media_platform.weibo.exception import DataFetchError

@retry(stop=stop_after_attempt(5), wait=wait_fixed(3), reraise=True)
async def weibo_search(client, uri, params):
    try:
        return await client.get(uri, params=params)
    except DataFetchError as e:
        # client already refreshed cookies before raising; retry uses the new session
        raise

Prevention

When it happens

Trigger: Weibo search/note API calls returning HTML or an error body instead of JSON: HTTP 432 anti-crawl responses, login-wall HTML when cookies expired, or rate-limit pages. The playwright_page.goto + update_cookies refresh runs first, then DataFetchError('get response code error: <code>') is raised.

Common situations: Aggressive search crawling triggering Weibo's 432 protection; expired h5 cookies after long sessions; missing browser context so the self-healing goto itself fails; Weibo tightening anti-bot checks.

Related errors

AI-assisted analysis of NanmiCoder/MediaCrawler@d6f7c5bb90 (2026-08-15). Data as JSON: /api/errors/eb03a3f9fbf17c9e. Report an issue: GitHub.