What causes "unknown error" in NanmiCoder/MediaCrawler?

Inspect the logged res JSON to identify the actual ok value and msg, then map it in the request handler if it's a recoverable state

NanmiCoder/MediaCrawler · error · DataFetchError

unknown error

Error message

unknown error

What it means

Raised as DataFetchError by WeiboClient.request (media_platform/weibo/client.py:100) when the JSON body's ok field is neither 1 (success) nor 0 (known error) — an unrecognized application code. The message defaults to 'unknown error' when the payload lacks 'msg'. This branch exists because Weibo occasionally returns other ok codes (e.g. during maintenance or new anti-crawl states) and the client treats anything unexpected as a hard failure.

Source

Thrown at media_platform/weibo/client.py:100

            return response

        try:
            data: Dict = response.json()
        except json.decoder.JSONDecodeError:
            # issue: #771 Search API returns error 432, retry multiple times + update h5 cookies
            utils.logger.error(f"[WeiboClient.request] request {method}:{url} err code: {response.status_code} res:{response.text}")
            await self.playwright_page.goto(self._host)
            await asyncio.sleep(2)
            await self.update_cookies(browser_context=self.playwright_page.context)
            raise DataFetchError(f"get response code error: {response.status_code}")

        ok_code = data.get("ok")
        if ok_code == 0:  # response error
            utils.logger.error(f"[WeiboClient.request] request {method}:{url} err, res:{data}")
            raise DataFetchError(data.get("msg", "response error"))
        elif ok_code != 1:  # unknown error
            utils.logger.error(f"[WeiboClient.request] request {method}:{url} err, res:{data}")
            raise DataFetchError(data.get("msg", "unknown error"))
        else:  # response right
            return data.get("data", {})

    async def get(self, uri: str, params=None, headers=None, **kwargs) -> Union[Response, Dict]:
        final_uri = uri
        if isinstance(params, dict):
            final_uri = (f"{uri}?"
                         f"{urlencode(params)}")

        if headers is None:
            headers = self.headers
        return await self.request(method="GET", url=f"{self._host}{final_uri}", headers=headers, **kwargs)

    async def post(self, uri: str, data: dict) -> Dict:
        json_str = json.dumps(data, separators=(',', ':'), ensure_ascii=False)
        return await self.request(method="POST", url=f"{self._host}{uri}", data=json_str, headers=self.headers)

    async def pong(self) -> bool:

View on GitHub (pinned to d6f7c5bb90)

Solutions

Inspect the logged res JSON to identify the actual ok value and msg, then map it in the request handler if it's a recoverable state
Add tenacity retry on the calling method for transient unknown codes
Update the client to handle the new ok code explicitly if Weibo changed the contract
Re-login if the code turns out to be session-related

Defensive patterns

Strategy: try-catch

Try / catch

from media_platform.weibo.exception import DataFetchError
from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(min=2, max=20), reraise=True)
async def call_weibo(client, uri, params):
    try:
        return await client.get(uri, params=params)
    except DataFetchError as e:
        if str(e) == "unknown error":
            raise  # transient/unrecognized ok code — retry with backoff
        raise

Prevention

Inspect the logged ok value and msg to classify new Weibo response codes
Wrap batch crawls in per-item error handling plus bounded retries
Pin and monitor the client against Weibo API changes; update the ok-code map when envelopes change

When it happens

Trigger: A weibo API response with ok not in {0,1} — e.g. ok:2 or missing entirely with a different envelope shape; typically after Weibo-side changes, intermittent app-level states, or endpoints that return a non-standard envelope.

Common situations: Weibo API envelope changes after an update breaking the ok-code contract; hitting endpoints not covered by the client's assumed response shape; transient server states during high load.

Related errors

AI-assisted analysis of NanmiCoder/MediaCrawler@d6f7c5bb90 (2026-08-15). Data as JSON: /api/errors/5fef0f7e22e2767c. Report an issue: GitHub.