NanmiCoder/MediaCrawler · error · DataFetchError

response error

Error message

response error

What it means

Raised as DataFetchError by WeiboClient.request (media_platform/weibo/client.py:97) when the parsed JSON body has ok == 0 — Weibo h5 API's explicit application-level error code. The message defaults to 'response error' when the response omits a 'msg' field; the actual data payload is logged just above. This is a business-logic rejection (bad params, invalid session, resource not found), distinct from transport failures.

Solutions

1. Check the logged res payload for Weibo's own msg describing the application error
2. Filter target ids (remove deleted/private notes) before requesting
3. Re-login if msg indicates session/auth problems
4. Handle per-item DataFetchError in the crawler loop so one bad note doesn't abort the batch

from media_platform.weibo.exception import DataFetchError

try:
    data = await client.get(uri, params=params)
except DataFetchError as e:
    if str(e) in ("response error", "unknown error"):
        log_and_skip(uri)  # app-level rejection (deleted note, bad params); don't abort batch
        return None
    raise

When it happens

Trigger: Any weibo client get/post where the JSON contains ok:0 — e.g. requesting a note/user that was deleted, malformed params, or a session whose cookies are valid transport-wise but rejected by the app layer. If the payload has no 'msg', the generic 'response error' text is used.

Common situations: Deleted/private weibo posts still in crawl queues; parameter changes after Weibo API updates; cookie sessions valid enough to get JSON but lacking required auth scopes.

Related errors

• unknown error
• [WeiboLogin.begin] Invalid Login Type Currently only…
• [BaiduTieBaLogin.begin]Invalid Login Type Currently only…
• Can not extract Tieba creator portrait from url
• Creator API response does not contain user info

AI-assisted analysis of NanmiCoder/MediaCrawler@d6f7c5bb90 (2026-08-15). Data as JSON: /api/errors/9bf7657b686870fa. Report an issue: GitHub.

Appendix: source

Thrown at media_platform/weibo/client.py:97

            response = await client.request(method, url, timeout=self.timeout, **kwargs)

        if enable_return_response:
            return response

        try:
            data: Dict = response.json()
        except json.decoder.JSONDecodeError:
            # issue: #771 Search API returns error 432, retry multiple times + update h5 cookies
            utils.logger.error(f"[WeiboClient.request] request {method}:{url} err code: {response.status_code} res:{response.text}")
            await self.playwright_page.goto(self._host)
            await asyncio.sleep(2)
            await self.update_cookies(browser_context=self.playwright_page.context)
            raise DataFetchError(f"get response code error: {response.status_code}")

        ok_code = data.get("ok")
        if ok_code == 0:  # response error
            utils.logger.error(f"[WeiboClient.request] request {method}:{url} err, res:{data}")
            raise DataFetchError(data.get("msg", "response error"))
        elif ok_code != 1:  # unknown error
            utils.logger.error(f"[WeiboClient.request] request {method}:{url} err, res:{data}")
            raise DataFetchError(data.get("msg", "unknown error"))
        else:  # response right
            return data.get("data", {})

    async def get(self, uri: str, params=None, headers=None, **kwargs) -> Union[Response, Dict]:
        final_uri = uri
        if isinstance(params, dict):
            final_uri = (f"{uri}?"
                         f"{urlencode(params)}")

        if headers is None:
            headers = self.headers
        return await self.request(method="GET", url=f"{self._host}{final_uri}", headers=headers, **kwargs)

    async def post(self, uri: str, data: dict) -> Dict:
        json_str = json.dumps(data, separators=(',', ':'), ensure_ascii=False)

View on GitHub (pinned to d6f7c5bb90)