{"record":{"id":"15d273ba481fb1ec","repo":"NanmiCoder/MediaCrawler","slug":"get-weibo-detail-err-response-text","errorCode":null,"errorMessage":"get weibo detail err: {response.text}","messagePattern":"get weibo detail err: (.+?)","errorType":"exception","errorClass":"DataFetchError","httpStatus":null,"severity":"error","filePath":"media_platform/weibo/client.py","lineNumber":268,"sourceCode":"        res_sub_comments = []\n        for comment in comment_list:\n            sub_comments = comment.get(\"comments\")\n            if sub_comments and isinstance(sub_comments, list):\n                await callback(note_id, sub_comments)\n                res_sub_comments.extend(sub_comments)\n        return res_sub_comments\n\n    async def get_note_info_by_id(self, note_id: str) -> Dict:\n        \"\"\"\n        Get note details by note ID\n        :param note_id:\n        :return:\n        \"\"\"\n        url = f\"{self._host}/detail/{note_id}\"\n        async with make_async_client(proxy=self.proxy) as client:\n            response = await client.request(\"GET\", url, timeout=self.timeout, headers=self.headers)\n            if response.status_code != 200:\n                raise DataFetchError(f\"get weibo detail err: {response.text}\")\n            match = re.search(r'var \\$render_data = (\\[.*?\\])\\[0\\]', response.text, re.DOTALL)\n            if match:\n                render_data_json = match.group(1)\n                render_data_dict = json.loads(render_data_json)\n                note_detail = render_data_dict[0].get(\"status\")\n                note_item = {\"mblog\": note_detail}\n                return note_item\n            else:\n                utils.logger.info(f\"[WeiboClient.get_note_info_by_id] $render_data value not found\")\n                return dict()\n\n    async def get_note_image(self, image_url: str) -> bytes:\n        image_url = image_url[8:]  # Remove https://\n        sub_url = image_url.split(\"/\")\n        image_url = \"\"\n        for i in range(len(sub_url)):\n            if i == 1:\n                image_url += \"large/\"  # Get high-resolution images","sourceCodeStart":250,"sourceCodeEnd":286,"githubUrl":"https://github.com/NanmiCoder/MediaCrawler/blob/d6f7c5bb906b6dac40ddf343ef9e26438a3de092/media_platform/weibo/client.py#L250-L286","documentation":"Raised by WeiboClient.get_note_info_by_id when the weibo.com /detail/{note_id} HTML endpoint returns any non-200 HTTP status. The message embeds the raw response body, so the actual reason (anti-bot page, deleted post, rate-limit page) is inside response.text. It is a DataFetchError, the generic fetch-failure exception for the weibo platform module.","triggerScenarios":"GET {host}/detail/{note_id} with expired/invalid cookies, a deleted or hidden weibo status ID, or an anti-crawler 412/302 interstitial returned by weibo.com when the request lacks a logged-in session or hits rate limits.","commonSituations":"Crawler runs with stale cookies after WEIBO login expiry, note IDs collected earlier that were later deleted, or aggressive crawl speed triggering weibo risk control; also proxy IP blacklisted by weibo.","solutions":["Inspect response.text inside the message: a login redirect means cookies expired - re-login via 'python main.py --platform weibo --lt qrcode' or refresh WEIBO_COOKIES.","If the note was deleted/hidden, treat the error as non-retryable and skip that note_id in your caller.","Slow down CRAWLER_MAX_NOTES_COUNT / enable and configure the proxy pool (ENABLE_IP_PROXY) so requests come from rotating IPs.","If the note detail is optional, fall back to the data already obtained from the creator feed instead of the /detail page."],"exampleFix":"// before\nnote = await wb_client.get_note_info_by_id(note_id)\n// after\ntry:\n    note = await wb_client.get_note_info_by_id(note_id)\nexcept DataFetchError as e:\n    utils.logger.warning(f\"skip note {note_id}: {e}\")\n    note = {}","handlingStrategy":"try-catch","validationCode":"def note_id_is_plausible(note_id: str) -> bool:\n    return bool(note_id) and note_id.isdigit() and len(note_id) >= 8","typeGuard":null,"tryCatchPattern":"from media_platform.weibo.exception import DataFetchError\ntry:\n    note = await wb_client.get_note_info_by_id(note_id)\nexcept DataFetchError as e:\n    if \"login\" in str(e).lower() or \"passport\" in str(e):\n        raise  # session expired - stop and re-login\n    utils.logger.warning(f\"skip note {note_id}: {e}\")\n    note = {}","preventionTips":["Keep weibo cookies fresh; re-login before long crawls.","Enable the proxy pool so no single IP gets flagged.","Sanitize note IDs before requesting details.","Log response bodies on first failure to distinguish anti-bot from deleted content."],"tags":["weibo","http","anti-bot","data-fetch"],"backgroundTag":null,"analyzedSha":"d6f7c5bb906b6dac40ddf343ef9e26438a3de092","analyzedAt":"2026-08-15T01:39:07.505Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}