{"record":{"id":"eb03a3f9fbf17c9e","repo":"NanmiCoder/MediaCrawler","slug":"get-response-code-error-response-status-code","errorCode":null,"errorMessage":"get response code error: {response.status_code}","messagePattern":"get response code error: (.+?)","errorType":"exception","errorClass":"DataFetchError","httpStatus":null,"severity":"error","filePath":"media_platform/weibo/client.py","lineNumber":92,"sourceCode":"        # Check if proxy is expired before each request\n        await self._refresh_proxy_if_expired()\n\n        enable_return_response = kwargs.pop(\"return_response\", False)\n        async with make_async_client(proxy=self.proxy) as client:\n            response = await client.request(method, url, timeout=self.timeout, **kwargs)\n\n        if enable_return_response:\n            return response\n\n        try:\n            data: Dict = response.json()\n        except json.decoder.JSONDecodeError:\n            # issue: #771 Search API returns error 432, retry multiple times + update h5 cookies\n            utils.logger.error(f\"[WeiboClient.request] request {method}:{url} err code: {response.status_code} res:{response.text}\")\n            await self.playwright_page.goto(self._host)\n            await asyncio.sleep(2)\n            await self.update_cookies(browser_context=self.playwright_page.context)\n            raise DataFetchError(f\"get response code error: {response.status_code}\")\n\n        ok_code = data.get(\"ok\")\n        if ok_code == 0:  # response error\n            utils.logger.error(f\"[WeiboClient.request] request {method}:{url} err, res:{data}\")\n            raise DataFetchError(data.get(\"msg\", \"response error\"))\n        elif ok_code != 1:  # unknown error\n            utils.logger.error(f\"[WeiboClient.request] request {method}:{url} err, res:{data}\")\n            raise DataFetchError(data.get(\"msg\", \"unknown error\"))\n        else:  # response right\n            return data.get(\"data\", {})\n\n    async def get(self, uri: str, params=None, headers=None, **kwargs) -> Union[Response, Dict]:\n        final_uri = uri\n        if isinstance(params, dict):\n            final_uri = (f\"{uri}?\"\n                         f\"{urlencode(params)}\")\n\n        if headers is None:","sourceCodeStart":74,"sourceCodeEnd":110,"githubUrl":"https://github.com/NanmiCoder/MediaCrawler/blob/d6f7c5bb906b6dac40ddf343ef9e26438a3de092/media_platform/weibo/client.py#L74-L110","documentation":"Raised as DataFetchError by WeiboClient.request (media_platform/weibo/client.py:92) when response.json() throws JSONDecodeError — the Weibo h5 API returned a non-JSON body, classically HTTP 432 on the search endpoint (issue #771). Before raising, the handler attempts self-healing: it navigates the Playwright page to the host and refreshes cookies, expecting upstream retries (tenacity on the caller) to succeed with the new session. So hitting this error means even that refresh path was entered and the raise still propagated.","triggerScenarios":"Weibo search/note API calls returning HTML or an error body instead of JSON: HTTP 432 anti-crawl responses, login-wall HTML when cookies expired, or rate-limit pages. The playwright_page.goto + update_cookies refresh runs first, then DataFetchError('get response code error: <code>') is raised.","commonSituations":"Aggressive search crawling triggering Weibo's 432 protection; expired h5 cookies after long sessions; missing browser context so the self-healing goto itself fails; Weibo tightening anti-bot checks.","solutions":["Rely on the built-in retry: callers decorated with tenacity retry will re-run with refreshed cookies — ensure the calling method has retry enabled","Slow down search request frequency (CRAWLER_MAX_SLEEP_SEC) to stop triggering 432","Re-login (qrcode/cookie) to get fresh h5 cookies if retries keep failing","Verify playwright_page is alive so the cookie-refresh goto can actually run"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"from tenacity import retry, stop_after_attempt, wait_fixed\nfrom media_platform.weibo.exception import DataFetchError\n\n@retry(stop=stop_after_attempt(5), wait=wait_fixed(3), reraise=True)\nasync def weibo_search(client, uri, params):\n    try:\n        return await client.get(uri, params=params)\n    except DataFetchError as e:\n        # client already refreshed cookies before raising; retry uses the new session\n        raise","preventionTips":["Keep tenacity retry on caller methods — the client self-heals cookies before raising","Throttle search frequency to avoid HTTP 432 anti-crawl responses","Keep playwright_page alive so the built-in goto+update_cookies refresh can run"],"tags":["weibo","anti-crawl","http-432","retry","cookies"],"backgroundTag":null,"analyzedSha":"d6f7c5bb906b6dac40ddf343ef9e26438a3de092","analyzedAt":"2026-08-15T01:39:07.505Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}