NanmiCoder/MediaCrawler · error · DataFetchError

get weibo detail err: {response.text}

Error message

get weibo detail err: {response.text}

What it means

Raised by WeiboClient.get_note_info_by_id when the weibo.com /detail/{note_id} HTML endpoint returns any non-200 HTTP status. The message embeds the raw response body, so the actual reason (anti-bot page, deleted post, rate-limit page) is inside response.text. It is a DataFetchError, the generic fetch-failure exception for the weibo platform module.

Source

Thrown at media_platform/weibo/client.py:268

        res_sub_comments = []
        for comment in comment_list:
            sub_comments = comment.get("comments")
            if sub_comments and isinstance(sub_comments, list):
                await callback(note_id, sub_comments)
                res_sub_comments.extend(sub_comments)
        return res_sub_comments

    async def get_note_info_by_id(self, note_id: str) -> Dict:
        """
        Get note details by note ID
        :param note_id:
        :return:
        """
        url = f"{self._host}/detail/{note_id}"
        async with make_async_client(proxy=self.proxy) as client:
            response = await client.request("GET", url, timeout=self.timeout, headers=self.headers)
            if response.status_code != 200:
                raise DataFetchError(f"get weibo detail err: {response.text}")
            match = re.search(r'var \$render_data = (\[.*?\])\[0\]', response.text, re.DOTALL)
            if match:
                render_data_json = match.group(1)
                render_data_dict = json.loads(render_data_json)
                note_detail = render_data_dict[0].get("status")
                note_item = {"mblog": note_detail}
                return note_item
            else:
                utils.logger.info(f"[WeiboClient.get_note_info_by_id] $render_data value not found")
                return dict()

    async def get_note_image(self, image_url: str) -> bytes:
        image_url = image_url[8:]  # Remove https://
        sub_url = image_url.split("/")
        image_url = ""
        for i in range(len(sub_url)):
            if i == 1:
                image_url += "large/"  # Get high-resolution images

View on GitHub (pinned to d6f7c5bb90)

Solutions

  1. Inspect response.text inside the message: a login redirect means cookies expired - re-login via 'python main.py --platform weibo --lt qrcode' or refresh WEIBO_COOKIES.
  2. If the note was deleted/hidden, treat the error as non-retryable and skip that note_id in your caller.
  3. Slow down CRAWLER_MAX_NOTES_COUNT / enable and configure the proxy pool (ENABLE_IP_PROXY) so requests come from rotating IPs.
  4. If the note detail is optional, fall back to the data already obtained from the creator feed instead of the /detail page.

Example fix

// before
note = await wb_client.get_note_info_by_id(note_id)
// after
try:
    note = await wb_client.get_note_info_by_id(note_id)
except DataFetchError as e:
    utils.logger.warning(f"skip note {note_id}: {e}")
    note = {}
Defensive patterns

Strategy: try-catch

Validate before calling

def note_id_is_plausible(note_id: str) -> bool:
    return bool(note_id) and note_id.isdigit() and len(note_id) >= 8

Try / catch

from media_platform.weibo.exception import DataFetchError
try:
    note = await wb_client.get_note_info_by_id(note_id)
except DataFetchError as e:
    if "login" in str(e).lower() or "passport" in str(e):
        raise  # session expired - stop and re-login
    utils.logger.warning(f"skip note {note_id}: {e}")
    note = {}

Prevention

When it happens

Trigger: GET {host}/detail/{note_id} with expired/invalid cookies, a deleted or hidden weibo status ID, or an anti-crawler 412/302 interstitial returned by weibo.com when the request lacks a logged-in session or hits rate limits.

Common situations: Crawler runs with stale cookies after WEIBO login expiry, note IDs collected earlier that were later deleted, or aggressive crawl speed triggering weibo risk control; also proxy IP blacklisted by weibo.

Related errors

AI-assisted analysis of NanmiCoder/MediaCrawler@d6f7c5bb90 (2026-08-15). Data as JSON: /api/errors/15d273ba481fb1ec. Report an issue: GitHub.