NanmiCoder/MediaCrawler · error

playwright_page is required for browser-based sub-comment fe

Error message

playwright_page is required for browser-based sub-comment fetching

What it means

Raised by BaiduTieBaClient.get_comments_all_sub_comments (media_platform/tieba/client.py:537) when sub-comment crawling is enabled (config.ENABLE_GET_SUB_COMMENTS true) but self.playwright_page is None. Note the early return above: with the flag off the method returns [] silently; with it on and no browser page it raises. Sub-comment pages are fetched by browser navigation, hence the requirement.

Source

Thrown at media_platform/tieba/client.py:537

        crawl_interval: float = 1.0,
        callback: Optional[Callable] = None,
    ) -> List[TiebaComment]:
        """
        Get all sub-comments for specified comments (uses Playwright to access page, avoiding API detection)
        Args:
            comments: Comment list
            crawl_interval: Crawl delay interval in seconds
            callback: Callback function after one post crawl completes

        Returns:
            List[TiebaComment]: Sub-comment list
        """
        if not config.ENABLE_GET_SUB_COMMENTS:
            return []

        if not self.playwright_page:
            utils.logger.error("[BaiduTieBaClient.get_comments_all_sub_comments] playwright_page is None, cannot use browser mode")
            raise Exception("playwright_page is required for browser-based sub-comment fetching")

        all_sub_comments: List[TiebaComment] = []

        for parment_comment in comments:
            if parment_comment.sub_comment_count == 0:
                continue

            current_page = 1
            max_sub_page_num = parment_comment.sub_comment_count // 10 + 1

            while max_sub_page_num >= current_page:
                # Construct sub-comment URL
                sub_comment_url = (
                    f"{self._host}/p/comment?"
                    f"tid={parment_comment.note_id}&"
                    f"pid={parment_comment.comment_id}&"
                    f"fid={parment_comment.tieba_id}&"
                    f"pn={current_page}"

View on GitHub (pinned to d6f7c5bb90)

Solutions

  1. Attach a Playwright page to the client when ENABLE_GET_SUB_COMMENTS is enabled
  2. Or set ENABLE_GET_SUB_COMMENTS=false if sub-comments are not required in a browserless run
  3. Ensure browser bootstrap completes before the comment-processing stage that calls this method
Defensive patterns

Strategy: validation

Validate before calling

import config
if config.ENABLE_GET_SUB_COMMENTS and not client.playwright_page:
    raise RuntimeError("ENABLE_GET_SUB_COMMENTS requires a browser page; disable it or attach playwright_page")

Type guard

def can_fetch_sub_comments(client) -> bool:
    import config
    return (not config.ENABLE_GET_SUB_COMMENTS) or (
        client.playwright_page is not None and not client.playwright_page.is_closed()
    )

Prevention

When it happens

Trigger: ENABLE_GET_SUB_COMMENTS=true combined with a client lacking playwright_page; any call to get_comments_all_sub_comments(comments, ...) under those conditions raises before iterating comments.

Common situations: Turning on sub-comment crawling in a pipeline that previously ran browserless; browser closed between note crawl and sub-comment crawl; tests enabling the flag without providing a page mock.

Related errors

AI-assisted analysis of NanmiCoder/MediaCrawler@d6f7c5bb90 (2026-08-15). Data as JSON: /api/errors/ab5af8eb3ed8a97a. Report an issue: GitHub.