NanmiCoder/MediaCrawler · error · DataFetchError

Get creator info error

Error message

Get creator info error

What it means

Raised by WeiboCrawler.get_creators_and_notes when get_creator_info_by_id returned a truthy response but its 'userInfo' key is missing or empty. It signals that the weibo user-info API answered without user data - typically because the UID does not exist, was banned, or risk control suppressed the payload. It is a DataFetchError thrown at the orchestration layer.

Source

Thrown at media_platform/weibo/core.py:316

            utils.logger.info(f"[WeiboCrawler.get_note_images] Sleeping for {config.CRAWLER_MAX_SLEEP_SEC} seconds after fetching image")
            if content != None:
                extension_file_name = url.split(".")[-1]
                await weibo_store.update_weibo_note_image(pid, content, extension_file_name)

    async def get_creators_and_notes(self) -> None:
        """
        Get creator's information and their notes and comments
        Returns:

        """
        utils.logger.info("[WeiboCrawler.get_creators_and_notes] Begin get weibo creators")
        for user_id in config.WEIBO_CREATOR_ID_LIST:
            createor_info_res: Dict = await self.wb_client.get_creator_info_by_id(creator_id=user_id)
            if createor_info_res:
                createor_info: Dict = createor_info_res.get("userInfo", {})
                utils.logger.info(f"[WeiboCrawler.get_creators_and_notes] creator info: {createor_info}")
                if not createor_info:
                    raise DataFetchError("Get creator info error")
                await weibo_store.save_creator(user_id, user_info=createor_info)

                # Create a wrapper callback to get full text before saving data
                async def save_notes_with_full_text(note_list: List[Dict]):
                    # If full text fetching is enabled, batch get full text first
                    updated_note_list = await self.batch_get_notes_full_text(note_list)
                    await weibo_store.batch_update_weibo_notes(updated_note_list)

                # Get all note information of the creator
                all_notes_list = await self.wb_client.get_all_notes_by_creator_id(
                    creator_id=user_id,
                    container_id=f"107603{user_id}",
                    crawl_interval=0,
                    callback=save_notes_with_full_text,
                )

                note_ids = [note_item.get("mblog", {}).get("id") for note_item in all_notes_list if note_item.get("mblog", {}).get("id")]
                await self.batch_get_notes_comments(note_ids)

View on GitHub (pinned to d6f7c5bb90)

Solutions

  1. Check the logged creator info line just above the raise to see what the API actually returned.
  2. Confirm each UID in WEIBO_CREATOR_ID_LIST resolves to a live profile in a browser.
  3. If the empty-userInfo case should not kill the run, catch DataFetchError around the loop body and continue with the next creator.
  4. Refresh login cookies / proxy, since risk-controlled sessions also return empty userInfo.

Example fix

// before
if not createor_info:
    raise DataFetchError("Get creator info error")
// after
if not createor_info:
    utils.logger.warning(f"creator {user_id} returned no userInfo, skipping")
    continue
Defensive patterns

Strategy: validation

Validate before calling

def creator_info_is_usable(createor_info_res: Dict) -> bool:
    return bool(createor_info_res.get("userInfo"))

Type guard

def is_creator_info(value: Dict) -> bool:
    info = value.get("userInfo") if isinstance(value, dict) else None
    return isinstance(info, dict) and len(info) > 0

Try / catch

try:
    res = await self.wb_client.get_creator_info_by_id(creator_id=user_id)
except DataFetchError as e:
    utils.logger.warning(f"creator {user_id} failed: {e}")
    continue

Prevention

When it happens

Trigger: A UID in WEIBO_CREATOR_ID_LIST that weibo's API does not return userInfo for; the response shape changed (API break); or an anti-bot JSON body without user data.

Common situations: Typo'd or outdated creator IDs pasted into config/base_config.py; account deactivated between crawl runs; weibo API contract change after a site update.

Related errors

AI-assisted analysis of NanmiCoder/MediaCrawler@d6f7c5bb90 (2026-08-15). Data as JSON: /api/errors/f707608ac6ebace5. Report an issue: GitHub.