NanmiCoder/MediaCrawler · error · DataFetchError

response error

Error message

response error

What it means

Raised as DataFetchError by WeiboClient.request (media_platform/weibo/client.py:97) when the parsed JSON body has ok == 0 — Weibo h5 API's explicit application-level error code. The message defaults to 'response error' when the response omits a 'msg' field; the actual data payload is logged just above. This is a business-logic rejection (bad params, invalid session, resource not found), distinct from transport failures.

Source

Thrown at media_platform/weibo/client.py:97

            response = await client.request(method, url, timeout=self.timeout, **kwargs)

        if enable_return_response:
            return response

        try:
            data: Dict = response.json()
        except json.decoder.JSONDecodeError:
            # issue: #771 Search API returns error 432, retry multiple times + update h5 cookies
            utils.logger.error(f"[WeiboClient.request] request {method}:{url} err code: {response.status_code} res:{response.text}")
            await self.playwright_page.goto(self._host)
            await asyncio.sleep(2)
            await self.update_cookies(browser_context=self.playwright_page.context)
            raise DataFetchError(f"get response code error: {response.status_code}")

        ok_code = data.get("ok")
        if ok_code == 0:  # response error
            utils.logger.error(f"[WeiboClient.request] request {method}:{url} err, res:{data}")
            raise DataFetchError(data.get("msg", "response error"))
        elif ok_code != 1:  # unknown error
            utils.logger.error(f"[WeiboClient.request] request {method}:{url} err, res:{data}")
            raise DataFetchError(data.get("msg", "unknown error"))
        else:  # response right
            return data.get("data", {})

    async def get(self, uri: str, params=None, headers=None, **kwargs) -> Union[Response, Dict]:
        final_uri = uri
        if isinstance(params, dict):
            final_uri = (f"{uri}?"
                         f"{urlencode(params)}")

        if headers is None:
            headers = self.headers
        return await self.request(method="GET", url=f"{self._host}{final_uri}", headers=headers, **kwargs)

    async def post(self, uri: str, data: dict) -> Dict:
        json_str = json.dumps(data, separators=(',', ':'), ensure_ascii=False)

View on GitHub (pinned to d6f7c5bb90)

Solutions

1. Check the logged res payload for Weibo's own msg describing the application error
2. Filter target ids (remove deleted/private notes) before requesting
3. Re-login if msg indicates session/auth problems
4. Handle per-item DataFetchError in the crawler loop so one bad note doesn't abort the batch

from media_platform.weibo.exception import DataFetchError

try:
    data = await client.get(uri, params=params)
except DataFetchError as e:
    if str(e) in ("response error", "unknown error"):
        log_and_skip(uri)  # app-level rejection (deleted note, bad params); don't abort batch
        return None
    raise

When it happens

Trigger: Any weibo client get/post where the JSON contains ok:0 — e.g. requesting a note/user that was deleted, malformed params, or a session whose cookies are valid transport-wise but rejected by the app layer. If the payload has no 'msg', the generic 'response error' text is used.

Common situations: Deleted/private weibo posts still in crawl queues; parameter changes after Weibo API updates; cookie sessions valid enough to get JSON but lacking required auth scopes.

Related errors

• unknown error
• [WeiboLogin.begin] Invalid Login Type Currently only support
• Invalid JSON file
• Wrong time range, please check your start and end argument,
• Unable to parse video ID from URL: {url}

AI-assisted analysis of NanmiCoder/MediaCrawler@d6f7c5bb90 (2026-08-15). Data as JSON: /api/errors/9bf7657b686870fa. Report an issue: GitHub.